A powerful TypeScript library for unified data integration across multiple sources
DataBinder simplifies the complexity of working with multiple APIs and data sources by providing a unified interface, robust error handling, intelligent retry logic, and enterprise-grade features like persistence, validation, and observability.
- 🔗 Universal Data Integration - Connect to REST APIs, GitHub, Microsoft Graph, OwnCloud/WebDAV, and more
- 📦 Datasource Catalog - Register, manage, and persist datasource configurations
- 🚀 Flexible Response Formats - Full, batch, iterator, and stream processing
- 🔄 Smart Retry Logic - Exponential backoff with configurable retry conditions
- 🛡️ Built-in Security - Input validation, sanitization, and authentication handling
- 📊 Observability Ready - OpenTelemetry integration with automatic instrumentation
- 💾 Persistence Layer - Database and file-based storage for datasource instances
- 🎯 Property Mapping - Transform and normalize data across different sources
- 🔍 Method Introspection - Discover, validate, and inspect datasource methods at runtime
- 📄 Document Parsing - Extract content from PDFs, Word documents, and more via OwnCloud
- 📄 Comprehensive Documentation - Detailed guides and examples for every feature
npm install @statuscompliance/databinderimport { DataBinder, Linker, DatasourceCatalog } from '@statuscompliance/databinder';
import { RestApiDatasource } from '@statuscompliance/databinder/Datasources';
// 1. Create catalog and register datasources
const catalog = new DatasourceCatalog();
catalog.registerDatasource(RestApiDatasource);
// 2. Create datasource instance
const apiInstance = catalog.createDatasourceInstance('rest-api', {
baseUrl: 'https://api.example.com',
auth: { type: 'bearer', token: 'your-token' }
}, 'my-api');
// 3. Create linker with datasource configuration
const linker = new Linker({
datasources: [apiInstance],
datasourceConfigs: {
'my-api': {
id: 'my-api',
methodConfig: {
methodName: 'default',
options: { endpoint: '/users' }
}
}
}
});
const dataBinder = new DataBinder({ linker });
// 4. Fetch data (uses configured method)
const data = await dataBinder.fetchAll();
// Or call a specific method with custom options
const users = await dataBinder.fetchFromDatasource('my-api', {
methodName: 'default',
endpoint: '/users'
});| Topic | Description | Link |
|---|---|---|
| Core Functionality | DataBinder and Linker classes | 📖 Read More |
| Datasource Catalog | Registration, persistence, and management | 📖 Read More |
| Built-in Datasources | REST API, GitHub, Microsoft Graph | 📖 Read More |
| Utilities | Validation, sanitization, logging, telemetry | 📖 Read More |
Generic REST API integration with full HTTP method support, authentication, and error handling.
const restApi = catalog.createDatasourceInstance('rest-api', {
baseUrl: 'https://api.example.com',
auth: { type: 'bearer', token: 'token' },
headers: { 'Content-Type': 'application/json' }
}, 'my-rest-api');Specialized GitHub integration with pre-built methods for repositories, issues, pull requests, and more.
const githubApi = catalog.createDatasourceInstance('github-api', {
personalAccessToken: 'ghp_xxxxxxxxxxxx',
defaultOrg: 'your-org'
}, 'my-github');Access Microsoft 365 services including users, groups, mail, and SharePoint.
const graphApi = catalog.createDatasourceInstance('microsoft-graph', {
tenantId: 'tenant-id',
clientId: 'client-id',
clientSecret: 'client-secret'
}, 'my-graph');Connect to OwnCloud/Nextcloud servers with WebDAV support for file operations and document parsing.
const ownCloud = catalog.createDatasourceInstance('owncloud', {
baseUrl: 'https://cloud.example.com',
username: 'your-username',
password: 'your-password'
}, 'my-owncloud');DataBinder follows a modular architecture that promotes reusability and extensibility:
┌─────────────────┐ ┌─────────────────┐ ┌────────────────────────┐
│ DataBinder │◄──►│ Linker │◄──►│ Datasources │
│ │ │ │ │ • REST API │
│ • fetchAll() │ │ • Method config │ │ • GitHub API │
│ • fetchFrom() │ │ • Introspection │ │ • Microsoft Graph │
│ • Iterators │ │ • Validation │ │ • OwnCloud/WebDAV │
└─────────────────┘ └─────────────────┘ └────────────────────────┘
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌────────────────────────┐
│ Utilities │ │ Catalog │ │ Persistence │
│ • Validation │ │ • Registry │ │ • Database Adapter │
│ • Sanitization │ │ • Factory │ │ • File System │
│ • Retry Logic │ │ • Serialization │ │ • Metadata │
│ • Telemetry │ │ • Validation │ │ • Import/Export │
│ • Document Utils│ │ • Introspection │ │ │
└─────────────────┘ └─────────────────┘ └────────────────────────┘
// Process large datasets efficiently
const iterator = await dataBinder.fetchAll({
responseFormat: 'iterator',
batchSize: 100
});
for await (const batch of iterator) {
console.log(`Processing ${batch.data.length} items...`);
await processBatch(batch.data);
}import { OwnCloudDatasource } from '@statuscompliance/databinder/Datasources';
// Register and create OwnCloud instance
catalog.registerDatasource(OwnCloudDatasource);
const ownCloud = catalog.createDatasourceInstance('owncloud', {
baseUrl: 'https://cloud.example.com',
username: 'user',
password: 'pass'
}, 'my-owncloud');
// Configure linker with OwnCloud datasource
const linker = new Linker({
datasources: [ownCloud],
datasourceConfigs: {
'my-owncloud': {
id: 'my-owncloud',
methodConfig: {
methodName: 'listRecursive',
options: { path: '/', maxDepth: 3 }
}
}
}
});
// Fetch and parse documents
const document = await dataBinder.fetchFromDatasource('my-owncloud', {
methodName: 'getDocument',
path: '/Documents/report.pdf',
parseContent: true
});
console.log(document.content); // Extracted text content
console.log(document.metadata); // Document metadata (author, creation date, etc.)const linker = new Linker({
datasources: [apiInstance],
datasourceConfigs: {
'my-api': {
id: 'my-api',
methodConfig: {
methodName: 'default',
options: { endpoint: '/users' }
},
propertyMapping: {
'user_id': 'id',
'full_name': 'name',
'created_at': 'createdDate'
}
}
}
});import { DatabaseAdapter, SerializedDatasourceInstance } from '@statuscompliance/databinder';
// Create database adapter
class MongoDBAdapter implements DatabaseAdapter {
constructor(private db: any) {}
async save(instances: SerializedDatasourceInstance[]): Promise<void> {
const operations = instances.map(instance => ({
replaceOne: {
filter: { id: instance.id },
replacement: instance,
upsert: true
}
}));
await this.db.collection('datasources').bulkWrite(operations);
}
async load(): Promise<SerializedDatasourceInstance[]> {
return await this.db.collection('datasources').find({}).toArray();
}
// Optional methods for single instance operations
async saveOne(instance: SerializedDatasourceInstance): Promise<void> {
await this.db.collection('datasources').replaceOne(
{ id: instance.id },
instance,
{ upsert: true }
);
}
async loadOne(id: string): Promise<SerializedDatasourceInstance | null> {
return await this.db.collection('datasources').findOne({ id });
}
}
// Use with catalog
const dbAdapter = new MongoDBAdapter(mongoDb);
await catalog.saveToDatabaseAdapter(dbAdapter, true); // Include metadata
await catalog.loadFromDatabaseAdapter(dbAdapter);Discover available methods and validate options before execution:
// List all available methods in a datasource
const methods = linker.listMethods('my-api');
console.log(methods); // ['default', 'getById', 'search', ...]
// Get detailed information about a method
const methodInfo = linker.getMethodInfo('my-api', 'getById');
console.log(methodInfo.description);
console.log(methodInfo.requiredOptions);
// Validate options before calling a method
const validation = linker.validateMethodOptions('my-api', 'getById', {
id: '123',
endpoint: '/users'
});
if (!validation.success) {
console.error('Invalid options:', validation.errors);
}
// Get all methods metadata
const allMethods = linker.getAllMethodsInfo('my-api');const customDatasource: DatasourceDefinition = {
id: 'custom-api',
name: 'My Custom API',
configSchema: {
type: 'object',
properties: {
apiKey: { type: 'string' },
baseUrl: { type: 'string' }
},
required: ['apiKey', 'baseUrl']
},
createInstance: (config) => ({
id: '',
definitionId: 'custom-api',
config,
methods: {
getData: async (options) => {
// Your implementation
return { data: [] };
}
}
})
};
catalog.registerDatasource(customDatasource);- API Aggregation - Combine data from multiple APIs into unified responses
- Data Pipeline Integration - Connect various data sources in ETL processes
- Microservices Communication - Standardize inter-service data fetching
- Analytics Platform - Collect data from diverse sources for analysis
- Node-RED Integration - Persistent datasource instances for flow reusability
- Multi-tenant Applications - Isolated datasource configurations per tenant
- Document Management - Parse and extract content from cloud storage (OwnCloud/Nextcloud)
- Content Migration - Transfer and transform data between different systems
We welcome contributions! Please see our Contributing Guide for details.
Apache License 2.0 - see LICENSE file for details.
Apache License 2.0