# Data Modelling SDK
## Overview
The Data Modelling SDK is a Rust library that provides unified interfaces for data modeling operations across multiple platforms (native apps, WASM/web apps, and API backends). It enables importing from various formats (SQL, ODCS, JSON Schema, AVRO, Protobuf, CADS, ODPS, BPMN, DMN, OpenAPI), exporting to multiple formats, model validation, and storage abstraction for different environments.
**Repository**: https://github.com/OffeneDatenmodellierung/data-modelling-sdk
**License**: MIT
**Rust Edition**: 2024
**Version**: 2.0.4
## Architecture
The SDK is designed with a modular architecture centered around:
1. **Storage Backends**: Abstract storage operations (file system, browser storage, HTTP API)
2. **Database Backends**: High-performance query layer (DuckDB, PostgreSQL)
3. **Models**: Core data structures (Table, Column, Relationship, DataModel)
4. **Import/Export**: Format converters for various data contract formats
5. **Validation**: Table and relationship validation logic
6. **Model Management**: Loading and saving models from storage backends
7. **Sync Engine**: Bidirectional YAML ↔ Database synchronization
8. **Decision Records (DDL)**: Architecture Decision Records with MADR format support
9. **Knowledge Base (KB)**: Domain documentation and knowledge articles
10. **Staging**: JSON data ingestion with DuckDB and Apache Iceberg support
11. **Inference**: Automatic schema inference from JSON data with format detection
12. **Mapping**: Schema-to-schema mapping with fuzzy matching and transformation generation
13. **Pipeline**: End-to-end data pipeline orchestration with checkpointing
14. **LLM**: LLM-enhanced schema refinement and matching (Ollama, llama.cpp)
## Directory Structure
```
data-modelling-sdk/
├── src/
│ ├── lib.rs # Main library entry point, re-exports public API
│ ├── auth/ # Authentication types (OAuth, session management)
│ │ └── mod.rs
│ ├── export/ # Export functionality
│ │ ├── mod.rs
│ │ ├── avro.rs # AVRO format exporter
│ │ ├── bpmn.rs # BPMN 2.0 exporter (feature-gated)
│ │ ├── cads.rs # CADS format exporter
│ │ ├── decision.rs # Decision Markdown exporter
│ │ ├── dmn.rs # DMN 1.3 exporter (feature-gated)
│ │ ├── json_schema.rs # JSON Schema exporter
│ │ ├── knowledge.rs # Knowledge article Markdown exporter
│ │ ├── odcl.rs # ODCL format exporter
│ │ ├── odcs.rs # ODCS v3.1.0 format exporter
│ │ ├── odps.rs # ODPS format exporter
│ │ ├── openapi.rs # OpenAPI 3.1.1 exporter (feature-gated)
│ │ ├── png.rs # PNG diagram exporter (feature-gated)
│ │ ├── protobuf.rs # Protobuf exporter
│ │ └── sql.rs # SQL DDL exporter
│ ├── convert/ # Universal format converter
│ │ ├── mod.rs
│ │ ├── converter.rs # Universal converter (any format → ODCS)
│ │ ├── migrate_dataflow.rs # DataFlow → Domain migration utility
│ │ └── openapi_to_odcs.rs # OpenAPI to ODCS converter (feature-gated)
│ ├── git/ # Git operations (feature-gated)
│ │ ├── mod.rs
│ │ └── git_service.rs # Git service for version control operations
│ ├── import/ # Import functionality
│ │ ├── mod.rs
│ │ ├── avro.rs # AVRO format importer
│ │ ├── bpmn.rs # BPMN 2.0 importer (feature-gated)
│ │ ├── cads.rs # CADS format importer
│ │ ├── decision.rs # Decision YAML importer
│ │ ├── dmn.rs # DMN 1.3 importer (feature-gated)
│ │ ├── json_schema.rs # JSON Schema importer
│ │ ├── knowledge.rs # Knowledge article YAML importer
│ │ ├── odcs.rs # ODCS v3.1.0 format importer (primary, also handles ODCL)
│ │ ├── odps.rs # ODPS format importer
│ │ ├── openapi.rs # OpenAPI 3.1.1 importer (feature-gated)
│ │ ├── protobuf.rs # Protobuf importer
│ │ └── sql.rs # SQL DDL importer
│ ├── model/ # Model loading/saving
│ │ ├── mod.rs
│ │ ├── api_loader.rs # API-based model loader
│ │ ├── loader.rs # File-based model loader
│ │ └── saver.rs # Model saver
│ ├── models/ # Core data structures
│ │ ├── mod.rs
│ │ ├── bpmn.rs # BPMN model structs (BPMNModel, BPMNModelFormat) (feature-gated)
│ │ ├── cads.rs # CADS model structs (CADSAsset, CADSKind, CADSBPMNModel, CADSDMNModel, CADSOpenAPISpec, etc.)
│ │ ├── column.rs # Column and ForeignKey models
│ │ ├── cross_domain.rs # Cross-domain relationship models
│ │ ├── data_model.rs # DataModel container (includes domain operations)
│ │ ├── decision.rs # Decision model structs (Decision, DecisionStatus, DecisionCategory, DecisionOption)
│ │ ├── dmn.rs # DMN model structs (DMNModel, DMNModelFormat) (feature-gated)
│ │ ├── domain.rs # Business Domain schema models (Domain, System, CADSNode, ODCSNode, etc.)
│ │ ├── enums.rs # Enums (DatabaseType, MedallionLayer, InfrastructureType, etc.)
│ │ ├── knowledge.rs # Knowledge model structs (KnowledgeArticle, ArticleType, ArticleStatus)
│ │ ├── odcs/ # ODCS native data structures (v2 API)
│ │ │ ├── mod.rs # Module exports
│ │ │ ├── contract.rs # ODCSContract - root contract document
│ │ │ ├── schema.rs # SchemaObject - schema/table definitions
│ │ │ ├── property.rs # Property - column/field with nested support
│ │ │ ├── supporting.rs # QualityRule, CustomProperty, Team, Server, etc.
│ │ │ └── converters.rs # From implementations for Table/Column interop
│ │ ├── odps.rs # ODPS model structs (ODPSDataProduct, ODPSInputPort, etc.)
│ │ ├── openapi.rs # OpenAPI model structs (OpenAPIModel, OpenAPIFormat) (feature-gated)
│ │ ├── relationship.rs # Relationship model
│ │ ├── table.rs # Table model
│ │ ├── tag.rs # Enhanced Tag enum (Simple, Pair, List)
│ │ └── workspace.rs # Workspace model with embedded relationships
│ ├── storage/ # Storage backend abstraction
│ │ ├── mod.rs # StorageBackend trait
│ │ ├── api.rs # HTTP API backend (feature-gated)
│ │ ├── browser.rs # Browser storage backend (WASM, feature-gated)
│ │ └── filesystem.rs # File system backend (native, feature-gated)
│ ├── database/ # Database backend abstraction
│ │ ├── mod.rs # DatabaseBackend trait, error types
│ │ ├── config.rs # Database configuration (.data-model.toml)
│ │ ├── schema.rs # Database schema definitions
│ │ ├── sync.rs # SyncEngine for YAML ↔ Database sync
│ │ ├── duckdb.rs # DuckDB backend (feature-gated)
│ │ └── postgres.rs # PostgreSQL backend (feature-gated)
│ ├── validation/ # Validation logic
│ │ ├── mod.rs
│ │ ├── input.rs # Input validation (table/column names, UUIDs, model names, file sizes)
│ │ ├── relationships.rs # Relationship validation (circular deps)
│ │ ├── tables.rs # Table validation (naming conflicts)
│ │ └── xml.rs # XML validation utilities (well-formedness checks)
│ ├── workspace/ # Workspace management types
│ │ └── mod.rs # WorkspaceInfo, ProfileInfo types
│ ├── staging/ # Data staging module (feature-gated: staging)
│ │ ├── mod.rs # Module exports
│ │ ├── config.rs # IngestConfig, DedupStrategy
│ │ ├── db.rs # StagingDb for DuckDB operations
│ │ ├── ingest.rs # JSON/JSONL ingestion logic
│ │ ├── batch.rs # Batch tracking and resume support
│ │ ├── catalog.rs # Iceberg catalog abstraction (REST, Unity, Glue, S3 Tables)
│ │ ├── iceberg_table.rs # Iceberg table operations
│ │ ├── export.rs # Export to production catalogs
│ │ └── error.rs # Staging error types
│ ├── inference/ # Schema inference module (feature-gated: inference)
│ │ ├── mod.rs # Module exports
│ │ ├── config.rs # InferenceConfig with builder pattern
│ │ ├── inferrer.rs # SchemaInferrer implementation
│ │ ├── types.rs # Type inference logic
│ │ ├── formats.rs # Format detection (18 formats)
│ │ ├── merge.rs # Schema merging and similarity
│ │ └── error.rs # Inference error types
│ ├── mapping/ # Schema mapping module (feature-gated: mapping)
│ │ ├── mod.rs # Module exports
│ │ ├── config.rs # MappingConfig, TransformFormat
│ │ ├── matcher.rs # SchemaMatcher with fuzzy matching
│ │ ├── types.rs # FieldMapping, TransformMapping, MatchMethod
│ │ ├── generator.rs # Transformation script generation (SQL, JQ, Python, PySpark)
│ │ ├── llm_matcher.rs # LLM-enhanced matching (feature-gated: llm)
│ │ └── error.rs # Mapping error types
│ ├── pipeline/ # Pipeline orchestration module (feature-gated: pipeline)
│ │ ├── mod.rs # Module exports
│ │ ├── config.rs # PipelineConfig, LlmPipelineConfig
│ │ ├── executor.rs # PipelineExecutor with stage execution
│ │ ├── checkpoint.rs # Checkpoint, StageOutput, PipelineStatus
│ │ └── error.rs # Pipeline error types
│ └── llm/ # LLM integration module (feature-gated: llm)
│ ├── mod.rs # Module exports
│ ├── config.rs # LlmMode, RefinementConfig
│ ├── client.rs # LlmClient trait, MockLlmClient
│ ├── ollama.rs # Ollama API client (feature-gated: llm-online)
│ ├── llamacpp.rs # llama.cpp client (feature-gated: llm-offline)
│ ├── prompt.rs # Prompt templates and context building
│ ├── validation.rs # Schema refinement validation
│ ├── docs.rs # Documentation loading (.txt, .md, .docx)
│ ├── refine.rs # SchemaRefiner implementation
│ └── error.rs # LLM error types
├── src/cli/commands/ # CLI command implementations
│ ├── mod.rs # Command module exports
│ ├── db.rs # Database commands (init, sync, status, export)
│ ├── decision.rs # Decision commands (new, list, show, status, export)
│ ├── knowledge.rs # Knowledge commands (new, list, show, search, status, export)
│ └── query.rs # SQL query command
├── tests/ # Test suite
│ ├── auth_tests.rs
│ ├── cads_tests.rs # CADS import/export tests
│ ├── domain_tests.rs # Business Domain schema tests
│ ├── export_tests.rs
│ ├── git_tests.rs
│ ├── import_tests.rs
│ ├── integration_tests.rs # Includes DataFlow migration and universal converter tests
│ ├── model_tests.rs
│ ├── models_tests.rs
│ ├── nested_structures_tests.rs
│ ├── odcs_comprehensive_tests.rs
│ ├── odps_tests.rs # ODPS import/export tests
│ ├── storage_tests.rs
│ ├── tag_tests.rs # Enhanced tag support tests
│ ├── validation_tests.rs
│ └── workspace_tests.rs
├── schemas/ # Schema reference directory
│ ├── README.md # Schema documentation
│ ├── odcs-json-schema-v3.1.0.json # ODCS v3.1.0 JSON Schema
│ ├── odcl-json-schema-1.2.1.json # ODCL v1.2.1 JSON Schema (legacy)
│ ├── odps-json-schema-latest.json # ODPS JSON Schema
│ └── cads.schema.json # CADS v1.0 JSON Schema
├── docs/ # Documentation
│ └── SCHEMA_OVERVIEW.md # Comprehensive schema overview guide
├── specs/ # Specification documents
│ └── 003-odcs-field-preservation/ # Current feature spec
│ ├── schemas/ # Original schema location (also copied to root schemas/)
│ └── ...
├── Cargo.toml # Package manifest
├── cargo-audit.toml # Security audit configuration
├── .pre-commit-config.yaml # Pre-commit hooks configuration
└── README.md # User documentation
```
## Key Modules
### Storage Backends (`src/storage/`)
Abstracts file operations across different environments:
- **`StorageBackend` trait**: Common interface for all storage backends
- **`FileSystemStorageBackend`**: Native file system operations (requires `native-fs` feature)
- **`BrowserStorageBackend`**: Browser IndexedDB/localStorage (WASM, requires `wasm` feature)
- **`ApiStorageBackend`**: HTTP API backend (requires `api-backend` feature, default)
### Database Backends (`src/database/`)
High-performance database layer for large workspaces (10-100x faster than file operations):
- **`DatabaseBackend` trait**: Common interface for all database backends
- **`DuckDBBackend`**: Embedded analytical database (requires `duckdb-backend` feature)
- **`PostgresBackend`**: Server-based relational database (requires `postgres-backend` feature)
- **`SyncEngine`**: Bidirectional sync between YAML files and database
- **`DatabaseConfig`**: Configuration management (`.data-model.toml`)
- **`DatabaseSchema`**: Schema definitions for workspaces, tables, columns, relationships
### Models (`src/models/`)
Core data structures:
- **`Table`**: Represents a database table/data contract with columns, metadata, and relationships. Supports enhanced tags (Simple, Pair, List)
- **`Column`**: Column definition with complete ODCS v3.1.0 property support. Includes all fields: `id`, `name`, `businessName`, `description`, `dataType`, `physicalType`, `physicalName`, `logicalTypeOptions`, `primaryKey`, `primaryKeyPosition`, `unique`, `nullable`, `partitioned`, `partitionKeyPosition`, `clustered`, `classification`, `criticalDataElement`, `encryptedName`, `transformSourceObjects`, `transformLogic`, `transformDescription`, `examples`, `defaultValue`, `relationships`, `authoritativeDefinitions`, `quality`, `enumValues`, `tags`, `customProperties`
- **`ColumnData`**: Import-format mirror of Column struct, preserving all ODCS v3.1.0 fields through WASM layer
- **`LogicalTypeOptions`**: Type constraint options (minLength, maxLength, pattern, format, minimum, maximum, precision, scale)
- **`AuthoritativeDefinition`**: Authoritative definition reference (type, url)
- **`Relationship`**: Relationship between tables with crow's feet notation support. Fields: sourceTableId, targetTableId, sourceCardinality, targetCardinality, flowDirection, relationshipType, color, sourceHandle, targetHandle, drawioEdgeId, foreignKeyDetails, etlJobMetadata, sla, contactDetails, infrastructureType, visualMetadata, notes, owner
- **`DataModel`**: Container for tables, relationships, and domains. Includes filter methods and domain operations (`add_domain`, `add_system_to_domain`, etc.)
- **`Domain`**: Business domain container with systems, CADS nodes, ODCS nodes, and connections
- **`System`**: Physical infrastructure entity (Kafka, Cassandra, EKS, etc.) with DataFlow metadata (owner, SLA, contact_details, infrastructure_type, notes, version)
- **`CADSNode`**: Reference to CADS asset (AI/ML model, application, pipeline) with shared reference support
- **`ODCSNode`**: Reference to ODCS Table with shared reference support
- **`CADSAsset`**: CADS asset model (AIModel, MLPipeline, Application, etc.) with full CADS v1.0 support. Includes `bpmn_models`, `dmn_models`, and `openapi_specs` fields for referencing process/decision/API models.
- **`ODPSDataProduct`**: ODPS data product model with input/output ports, support, team, and custom properties
- **`BPMNModel`**: BPMN 2.0 process model stored in native XML format (requires `bpmn` feature)
- **`DMNModel`**: DMN 1.3 decision model stored in native XML format (requires `dmn` feature)
- **`Decision`**: Architecture Decision Record following MADR format with full lifecycle support
- **`DecisionStatus`**: Decision lifecycle states (Draft, Proposed, Accepted, Deprecated, Superseded, Rejected)
- **`DecisionCategory`**: Decision categories (Architecture, Technology, Process, Security, Data, Integration)
- **`DecisionOption`**: Decision option with title, description, pros/cons
- **`DecisionIndex`**: Index tracking decision metadata and numbering
- **`KnowledgeArticle`**: Knowledge base article with structured content and metadata
- **`ArticleType`**: Article types (Guide, Reference, Concept, Tutorial, Troubleshooting, Runbook)
- **`ArticleStatus`**: Article lifecycle states (Draft, Review, Published, Archived, Deprecated)
- **`KnowledgeIndex`**: Index tracking article metadata and numbering
- **`OpenAPIModel`**: OpenAPI 3.1.1 specification stored in native YAML or JSON format (requires `openapi` feature)
- **`CADSBPMNModel`**: CADS reference to BPMN model (name, reference, format, description)
- **`CADSDMNModel`**: CADS reference to DMN model (name, reference, format, description)
- **`CADSOpenAPISpec`**: CADS reference to OpenAPI spec (name, reference, format, description)
- **`Tag`**: Enhanced tag enum supporting Simple, Pair, and List formats
- **`Workspace`**: Workspace configuration containing domains, systems, assets, and embedded relationships. Helper methods: `add_relationship()`, `remove_relationship()`, `get_relationships_for_source()`, `get_relationships_for_target()`
- **`ForeignKey`**: Foreign key relationship details
- **`SlaProperty`**: SLA property structure (property, value, unit, element, driver, description, scheduler, schedule)
- **`ContactDetails`**: Contact details structure (email, phone, name, role, other)
- **Enums**: `DatabaseType`, `MedallionLayer`, `SCDPattern`, `DataVaultClassification`, `ModelingLevel`, `Cardinality` (oneToOne, oneToMany, manyToOne, manyToMany), `RelationshipType` (dataFlow, dependency, foreignKey, etl), `EndpointCardinality` (zeroOrOne, exactlyOne, zeroOrMany, oneOrMany), `FlowDirection` (sourceToTarget, targetToSource, bidirectional), `InfrastructureType` (70+ infrastructure types), `CADSKind`, `CADSStatus`, `ODPSStatus`
### ODCS Native Types (`src/models/odcs/`) - v2 API
Native ODCS v3.1.0 data structures for lossless round-trip import/export:
- **`ODCSContract`**: Root data contract document. Contains all contract-level fields: `apiVersion`, `kind`, `id`, `version`, `name`, `status`, `domain`, `dataProduct`, `tenant`, `description`, `schema` (Vec<SchemaObject>), `servers`, `team`, `support`, `roles`, `serviceLevels`, `quality`, `price`, `terms`, `links`, `tags`, `customProperties`
- **`SchemaObject`**: Schema/table definition within a contract. Contains: `name`, `physicalName`, `physicalType`, `businessName`, `description`, `dataGranularityDescription`, `properties` (Vec<Property>), `relationships`, `quality`, `authoritativeDefinitions`, `customProperties`
- **`Property`**: Column/field definition with nested support. Contains all ODCS property fields plus: `properties` (Vec<Property> for OBJECT types), `items` (Box<Property> for ARRAY types). Supports arbitrary nesting depth.
- **`QualityRule`**: Quality rule definition (type, column, description, mustBe, mustNotBe, etc.)
- **`CustomProperty`**: Key-value custom property (property, value)
- **`AuthoritativeDefinition`**: External definition reference (type, url)
- **`Team`**, **`TeamMember`**, **`Support`**, **`Server`**, **`Role`**, **`ServiceLevel`**, **`Price`**, **`Terms`**, **`Link`**: Supporting types for contract metadata
Converters for backwards compatibility:
- `From<&Property> for Column` - Flattens nested properties to dot-notation
- `From<&Column> for Property` - Creates flat property from column
- `From<&SchemaObject> for Table` - Converts schema to table with flattened columns
- `From<&Table> for SchemaObject` - Reconstructs schema from table
- `ODCSContract::to_tables()` - Converts contract to Vec<Table>
- `ODCSContract::from_tables()` - Creates contract from Vec<Table>
- `ODCSContract::to_table_data()` - Converts contract to Vec<TableData> for UI
### Import (`src/import/`)
Importers convert various formats to SDK models:
- **`ODCSImporter`**: Primary importer for ODCS v3.1.0 format (also handles legacy ODCL v1.2.1) - for Data Models (tables). Preserves `description`, `quality`, and `$ref` fields. Supports enhanced tags.
- `import()` - Returns flattened Table/Column types (v1 API)
- `import_contract()` - Returns native ODCSContract with nested properties (v2 API, lossless)
- **`CADSImporter`**: Importer for CADS v1.0 format - for compute assets (AI/ML models, applications, pipelines). Supports `bpmn_models`, `dmn_models`, and `openapi_specs` references.
- **`ODPSImporter`**: Importer for ODPS format - for data products linking to ODCS Tables
- **`BPMNImporter`**: Importer for BPMN 2.0 XML format - for business process models (requires `bpmn` feature)
- **`DMNImporter`**: Importer for DMN 1.3 XML format - for decision models (requires `dmn` feature)
- **`OpenAPIImporter`**: Importer for OpenAPI 3.1.1 YAML/JSON format - for API specifications (requires `openapi` feature)
- **`DecisionImporter`**: Importer for Decision YAML files - for architecture decision records
- **`KnowledgeImporter`**: Importer for Knowledge article YAML files - for domain documentation
- **`SQLImporter`**: SQL DDL parser (CREATE TABLE, CREATE VIEW, CREATE MATERIALIZED VIEW statements)
- Supported SQL dialects: PostgreSQL, MySQL, SQLite, Generic (default), and Databricks
- Databricks dialect supports: `IDENTIFIER()` function calls, variable references (`:variable_name`) in types/columns/metadata, `STRUCT`/`ARRAY` complex types, and views/materialized views
- **`JSONSchemaImporter`**: JSON Schema to Table conversion
- **`AvroImporter`**: AVRO schema to Table conversion
- **`ProtobufImporter`**: Protobuf .proto files to Table conversion (supports proto2 and proto3 syntax)
### Export (`src/export/`)
Exporters convert SDK models to various formats:
- **`ODCSExporter`**: Exports to ODCS v3.1.0 format - for Data Models (tables). Preserves all fields including `description`, `quality`, and `$ref`. Supports enhanced tags.
- `export_table()` - Exports Table with reconstruction from flattened columns (v1 API)
- `export_contract()` - Directly serializes ODCSContract (v2 API, lossless)
- `export_contract_validated()` - Same as export_contract() with optional schema validation
- **`ODCLExporter`**: Exports to ODCL v1.2.1 format - for legacy compatibility
- **`CADSExporter`**: Exports to CADS v1.0 format - for compute assets. Supports `bpmn_models`, `dmn_models`, and `openapi_specs` references.
- **`ODPSExporter`**: Exports to ODPS format - for data products
- **`BPMNExporter`**: Exports BPMN models in native XML format (requires `bpmn` feature)
- **`DMNExporter`**: Exports DMN models in native XML format (requires `dmn` feature)
- **`OpenAPIExporter`**: Exports OpenAPI specs with YAML ↔ JSON conversion support (requires `openapi` feature)
- **`DecisionExporter`**: Exports decisions to MADR-compliant Markdown format
- **`KnowledgeExporter`**: Exports knowledge articles to Markdown format
- **`SQLExporter`**: Generates SQL DDL (CREATE TABLE statements)
- Supports multiple SQL dialects: PostgreSQL, MySQL, SQLite, Generic, and Databricks
- **`JSONSchemaExporter`**: Exports to JSON Schema format
- **`AvroExporter`**: Exports to AVRO schema format
- **`ProtobufExporter`**: Exports to Protobuf .proto format (supports proto2 and proto3 via `export_with_version()`)
- **`PNGExporter`**: Generates PNG diagrams (requires `png-export` feature)
### Model Management (`src/model/`)
- **`ModelLoader`**: Loads models from storage backends. Supports both legacy `tables/` directory structure and new domain-based structure:
- `load_model()`: Legacy method for loading from `tables/` directory
- `load_domains()`: Loads all domains from domain directories (auto-discovery)
- `load_domains_from_list()`: Loads domains from explicit directory names
- `load_domain()`: Loads a single domain from a domain directory
- `load_domain_odcs_tables()`: Loads ODCS tables from a domain directory
- `load_domain_odps_products()`: Loads ODPS products from a domain directory
- `load_domain_cads_assets()`: Loads CADS assets from a domain directory
- `load_bpmn_models()`: Loads BPMN models from a domain directory (requires `bpmn` feature)
- `load_bpmn_model()`: Loads a specific BPMN model by name (requires `bpmn` feature)
- `load_bpmn_xml()`: Loads BPMN XML content (requires `bpmn` feature)
- `load_dmn_models()`: Loads DMN models from a domain directory (requires `dmn` feature)
- `load_dmn_model()`: Loads a specific DMN model by name (requires `dmn` feature)
- `load_dmn_xml()`: Loads DMN XML content (requires `dmn` feature)
- `load_openapi_models()`: Loads OpenAPI specs from a domain directory (requires `openapi` feature)
- `load_openapi_model()`: Loads a specific OpenAPI spec by name (requires `openapi` feature)
- `load_openapi_content()`: Loads OpenAPI content (requires `openapi` feature)
- `load_decisions()`: Loads all decisions from decisions/ directory
- `load_decision_index()`: Loads decision index metadata
- `load_decisions_by_domain()`: Loads decisions filtered by domain
- `load_knowledge()`: Loads all knowledge articles from knowledge/ directory
- `load_knowledge_index()`: Loads knowledge index metadata
- `load_knowledge_by_domain()`: Loads knowledge articles filtered by domain
- **`ModelSaver`**: Saves models to storage backends. Supports domain-based file structure:
- `save_table()`: Legacy method for saving to `tables/` directory
- `save_domain()`: Saves domain with all associated ODCS/ODPS/CADS files to domain directory
- `save_odps_product()`: Saves ODPS product to domain directory
- `save_cads_asset()`: Saves CADS asset to domain directory
- `save_bpmn_model()`: Saves BPMN model to domain directory (requires `bpmn` feature)
- `save_dmn_model()`: Saves DMN model to domain directory (requires `dmn` feature)
- `save_openapi_model()`: Saves OpenAPI spec to domain directory (requires `openapi` feature)
- `save_decision()`: Saves a decision to decisions/ directory
- `save_decision_index()`: Saves decision index metadata
- `save_knowledge()`: Saves a knowledge article to knowledge/ directory
- `save_knowledge_index()`: Saves knowledge index metadata
- `export_decision_markdown()`: Exports a decision to Markdown in decisions-md/
- `export_knowledge_markdown()`: Exports a knowledge article to Markdown in knowledge-md/
- `export_all_decisions_markdown()`: Exports all decisions to Markdown
- `export_all_knowledge_markdown()`: Exports all knowledge articles to Markdown
- **`ApiModelLoader`**: Loads models via HTTP API
- **`DomainLoadResult`**: Result structure containing loaded domains, tables, ODPS products, and CADS assets
### Converters (`src/convert/`)
- **`convert_to_odcs()`**: Universal converter that converts any supported format to ODCS v3.1.0 format. Supports auto-detection or explicit format specification. Formats: SQL, ODCS, ODCL, JSON Schema, AVRO, Protobuf, CADS, ODPS, Domain.
- **`OpenAPIToODCSConverter`**: Converts OpenAPI 3.1.1 schema components to ODCS table definitions (requires `openapi` feature)
- `convert_component()`: Converts a single OpenAPI component to an ODCS table
- `convert_components()`: Converts multiple OpenAPI components to ODCS tables
- `analyze_conversion()`: Analyzes conversion feasibility and returns a conversion report
- Type mapping: OpenAPI types (string, integer, number, boolean) mapped to ODCS types
- Constraint preservation: minLength, maxLength, pattern, minimum, maximum, enum converted to ODCS quality rules
- Format support: date, date-time, email, uri, uuid, password formats handled
- **`migrate_dataflow_to_domain()`**: Migrates legacy DataFlow YAML format to Domain schema format
### Validation (`src/validation/`)
- **`TableValidator`**: Validates table names, detects naming conflicts
- **`RelationshipValidator`**: Validates relationships, detects circular dependencies
- **`InputValidator`**: Validates table/column names, UUIDs, SQL identifiers, model names, file sizes
- **`XML validation utilities`**: Basic XML well-formedness checks for BPMN/DMN (requires `bpmn`/`dmn` features)
- **`DataModel` filter methods**: `filter_nodes_by_owner()`, `filter_relationships_by_owner()`, `filter_nodes_by_infrastructure_type()`, `filter_relationships_by_infrastructure_type()`, `filter_by_tags()` (supports enhanced tags)
- **`DataModel` domain methods**: `add_domain()`, `get_domain_by_id()`, `get_domain_by_name()`, `add_system_to_domain()`, `add_cads_node_to_domain()`, `add_odcs_node_to_domain()`, `add_system_connection_to_domain()`, `add_node_connection_to_domain()`
### Git Operations (`src/git/`)
- **`GitService`**: Git operations (status, commit, push, pull) - requires `git` feature
### Staging (`src/staging/`) - Feature: `staging`
Data ingestion and staging with DuckDB and Apache Iceberg support:
- **`StagingDb`**: DuckDB-based staging database
- `open(path)`: Open or create a staging database
- `memory()`: Create in-memory database for testing
- `init()`: Initialize database schema
- `record_count(partition)`: Count staged records
- `get_sample(limit, partition)`: Get sample records
- `query(sql)`: Execute arbitrary SQL
- `list_batches(limit)`: List batch history
- `partition_stats()`: Get partition statistics
- **`IngestConfig`**: Ingestion configuration
- `batch_size`: Insert batch size (default: 1000)
- `dedup_strategy`: Deduplication strategy (None, ByPath, ByContent, Both)
- `partition_key`: Optional partition key
- `file_pattern`: Glob pattern for file matching
- **`IcebergTable`**: Apache Iceberg table operations
- `create(catalog, schema)`: Create new Iceberg table
- `append_records(records)`: Append JSON records as Parquet
- `get_snapshot(version)`: Time travel by version
- `list_snapshots()`: List table history
- `get_batch_metadata()`: Get batch tracking metadata
- **`CatalogConfig`**: Iceberg catalog abstraction
- `Rest`: REST catalog (Lakekeeper, Nessie, Polaris)
- `Unity`: Databricks Unity Catalog
- `Glue`: AWS Glue Data Catalog
- `S3Tables`: AWS S3 Tables
- **`export_to_catalog()`**: Export to production catalogs
- **`IngestProgress`**: Real-time progress reporting (feature: `staging`)
- Multi-bar display with files, records, bytes throughput
- `new(total_files, show_bytes)`: Create progress display
- `start_file(path)`: Begin processing a file
- `add_records(count)`: Update record count
- `add_bytes(count)`: Update bytes processed
- `finish()`: Complete progress display
- **`InferenceProgress`**: Progress bar for schema inference
- `new(total_records)`: Create progress bar
- `inc(count)`: Increment progress
- `finish()`: Complete with message
- **`Spinner`**: Simple spinner for indeterminate operations
- `new(message)`: Create spinner
- `finish_with_message(message)`: Complete with status
- **`S3Ingester`**: AWS S3 ingestion (feature: `s3`)
- `new(source)`: Create with S3Source configuration
- `discover_files()`: List files matching pattern
- `download_file(key)`: Stream download file content
- Supports custom endpoints (MinIO, LocalStack)
- **`S3Source`**: S3 configuration
- `bucket`: S3 bucket name
- `prefix`: Optional key prefix filter
- `region`: AWS region
- `profile`: Optional AWS profile name
- `endpoint_url`: Optional custom endpoint
- **`UnityVolumeIngester`**: Databricks Unity Catalog Volumes (feature: `databricks`)
- `new(source)`: Create with UnityVolumeSource configuration
- `discover_files()`: List files in volume
- `download_file(path)`: Download file content via REST API
- **`UnityVolumeSource`**: Unity Catalog configuration
- `host`: Databricks workspace URL
- `token`: SecureCredentials wrapper for PAT
- `catalog`: Unity Catalog name
- `schema`: Schema name
- `volume`: Volume name
- `path`: Optional path within volume
- **`SecureCredentials`**: Credential wrapper preventing accidental logging
- Implements Display/Debug to show redacted value
- `new(value)`: Create secure wrapper
- `expose()`: Get actual credential value
- **`redact_secret(value, visible_chars)`**: Redact secret showing first N chars
- **`redact_secrets_in_string(text)`**: Regex-based redaction for:
- AWS access keys (AKIA...)
- AWS secret keys
- Bearer tokens
- URL-embedded passwords
### Inference (`src/inference/`) - Feature: `inference`
Automatic schema inference from JSON data:
- **`SchemaInferrer`**: Main inference engine
- `new()`: Create with default config
- `with_config(config)`: Create with custom config
- `add_json(json)`: Add single JSON record
- `add_value(value)`: Add parsed serde_json::Value
- `add_json_batch(records)`: Add batch of records
- `finalize()`: Generate InferredSchema
- `stats()`: Get inference statistics
- **`InferenceConfig`**: Configuration options
- `sample_size`: Max records to sample (0 = all)
- `min_field_frequency`: Min occurrence rate (0.0-1.0)
- `detect_formats`: Enable format detection
- `max_depth`: Max nested object depth
- `collect_examples`: Collect example values
- `max_examples`: Max examples per field
- **`InferredSchema`**: Inference result
- `to_json_schema()`: Convert to JSON Schema format
- Type support: null, boolean, integer, number, string, array, object, mixed
- **Format Detection** (18 formats):
- Temporal: `date-time`, `date`, `time`
- Network: `email`, `uri`, `hostname`, `ipv4`, `ipv6`
- Identifiers: `uuid`, `semver`, `slug`
- Standards: `country_code`, `currency_code`, `language_code`, `mime_type`
- Encoded: `base64`, `jwt`
- Contact: `phone` (E.164)
- **Schema Merging**:
- `merge_schemas()`: Merge compatible schemas
- `group_similar_schemas()`: Group by similarity (Jaccard index)
### Mapping (`src/mapping/`) - Feature: `mapping`
Schema-to-schema mapping with transformation generation:
- **`SchemaMatcher`**: Field matching engine
- `new()`: Create with default config
- `with_config(config)`: Create with custom config
- `match_schemas(source, target)`: Match two JSON Schemas
- Match methods: Exact, CaseInsensitive, Fuzzy (Levenshtein), Semantic, LLM
- **`MappingConfig`**: Configuration options
- `min_confidence`: Minimum match confidence (0.0-1.0)
- `fuzzy_matching`: Enable fuzzy matching
- `case_insensitive`: Enable case-insensitive matching
- `transform_format`: Output format (SQL, JQ, Python, PySpark)
- **`SchemaMapping`**: Mapping result
- `direct_mappings`: Direct field-to-field mappings
- `transformations`: Fields requiring transformation
- `gaps`: Target fields with no source
- `extras`: Source fields not mapped
- `compatibility_score`: Overall compatibility (0.0-1.0)
- **`FieldMapping`**: Individual field mapping
- `source_path`, `target_path`: Field paths
- `confidence`: Match confidence
- `type_compatible`: Whether types are compatible
- `match_method`: How the match was found
- **`TransformMapping`**: Transformation definition
- `source_paths`: Source field(s)
- `target_path`: Target field
- `transform_type`: TypeCast, Rename, Merge, Split, FormatChange, Custom, Extract, Default
- **`generate_transform()`**: Generate transformation scripts
- SQL: DuckDB-compatible INSERT/SELECT
- JQ: jq filter expressions
- Python: pandas transformation script
- PySpark: PySpark DataFrame transformation
- **`LlmSchemaMatcher`**: LLM-enhanced matching (feature: `llm`)
- Combines algorithmic and LLM-based matching
- Retry logic with exponential backoff
- Batching for large schemas
- Transform hint parsing
### Pipeline (`src/pipeline/`) - Feature: `pipeline`
End-to-end data pipeline orchestration:
- **`PipelineExecutor`**: Pipeline execution engine
- `new(config)`: Create with configuration
- `run()`: Execute full pipeline
- Returns `PipelineReport` with stage results
- **`PipelineConfig`**: Pipeline configuration
- `database_path`: Staging database path
- `source_path`: Input data source
- `output_dir`: Output directory
- `target_schema`: Optional target schema for mapping
- `stages`: Stages to execute
- `llm_config`: LLM configuration
- `dry_run`: Validate without executing
- `resume`: Resume from checkpoint
- **`PipelineStage`**: Pipeline stages
- `Ingest`: Load JSON into staging database
- `Infer`: Infer schema from staged data
- `Refine`: LLM-enhanced schema refinement
- `Map`: Map to target schema
- `Export`: Export results
- **`Checkpoint`**: Checkpoint for resume support
- `load(path)`: Load existing checkpoint
- `save(path)`: Save checkpoint
- `completed_stages`: List of completed stages
- `current_stage`: Currently executing stage
- `stage_outputs`: Results for each stage
- `config_hash`: Configuration hash for change detection
- **`PipelineReport`**: Execution report
- `stages`: Stage execution results
- `total_duration`: Total execution time
- `success`: Whether pipeline succeeded
- `print_summary()`: Print human-readable summary
### LLM (`src/llm/`) - Feature: `llm`
LLM integration for schema refinement:
- **`LlmClient` trait**: Common interface for LLM backends
- `complete(prompt)`: Send prompt and get completion
- `model_name()`: Get model name
- `max_tokens()`: Get max token limit
- `is_ready()`: Check if client is ready
- **`OllamaClient`**: Ollama API client (feature: `llm-online`)
- Connects to local or remote Ollama server
- Supports `/api/generate` endpoint
- Configurable model, temperature, timeout
- **`LlamaCppClient`**: llama.cpp client (feature: `llm-offline`)
- Local inference with GGUF models
- GPU acceleration support
- Configurable context size
- **`RefinementConfig`**: Refinement configuration
- `llm_mode`: None, Online, Offline
- `documentation_path`: Path to context documentation
- `temperature`: Generation temperature
- `max_retries`: Retry count on failure
- `include_samples`: Include sample records in prompt
- **`SchemaRefiner`**: Schema refinement engine
- `refine(schema, samples)`: Refine schema with LLM
- `RefinementResult`: Contains refined schema and metadata
- Validation ensures field preservation
- **`DocumentationLoader`**: Load context documentation
- Supports `.txt`, `.md`, `.docx` formats
- `extract_relevant_sections()`: Keyword-based extraction
- `truncate_documentation()`: Token-aware truncation
- **`ValidationResult`**: Refinement validation
- Ensures all original fields preserved
- Detects field renames (not allowed)
- Validates type compatibility
- Allows narrowing (number → integer)
## Features
The SDK uses Cargo features to enable optional functionality:
- **`default`**: Includes `api-backend` (HTTP API support)
- **`api-backend`**: Enables `ApiStorageBackend` (reqwest, urlencoding)
- **`native-fs`**: Enables `FileSystemStorageBackend` (tokio)
- **`wasm`**: Enables `BrowserStorageBackend` (wasm-bindgen, web-sys)
- **`png-export`**: Enables PNG diagram export (image, imageproc)
- **`databricks-dialect`**: Enables Databricks SQL dialect support (datafusion)
- **`git`**: Enables Git operations (git2)
- **`database`**: Enables database backend support
- **`duckdb-backend`**: Enables DuckDB embedded database (duckdb)
- **`postgres-backend`**: Enables PostgreSQL database (tokio-postgres)
- **`cli-full`**: Full CLI with all features including database support
- **`staging`**: Enables data staging module (duckdb, indicatif for progress)
- **`s3`**: Enables S3 ingestion (aws-config, aws-sdk-s3, aws-credential-types)
- **`databricks`**: Enables Databricks Unity Catalog Volumes ingestion (reqwest)
- **`inference`**: Enables schema inference module
- **`mapping`**: Enables schema mapping module (depends on `inference`)
- **`pipeline`**: Enables pipeline orchestration (depends on `staging`, `inference`, `mapping`)
- **`llm`**: Enables LLM integration base (tokio, zip)
- **`llm-online`**: Enables Ollama API client (depends on `llm`, adds `reqwest`)
- **`llm-offline`**: Enables llama.cpp client (depends on `llm`)
- **`iceberg`**: Enables Apache Iceberg support (iceberg, iceberg-catalog-rest, arrow, parquet)
## Dependencies
### Core Dependencies
- `serde`, `serde_json`, `serde_yaml`: Serialization
- `anyhow`, `thiserror`: Error handling
- `async-trait`: Async trait support
- `uuid`: UUID generation (v4 random, v5 deterministic)
- `chrono`: Timestamps
- `tracing`: Logging
- `petgraph`: Graph operations for validation
### Optional Dependencies
- `tokio`: Async runtime (for `native-fs` feature)
- `reqwest`: HTTP client (for `api-backend` feature)
- `wasm-bindgen`, `web-sys`: WASM support (for `wasm` feature)
- `git2`: Git operations (for `git` feature)
- `image`, `imageproc`: Image processing (for `png-export` feature)
- `datafusion`: SQL parsing (for `databricks-dialect` feature)
- `duckdb`: DuckDB embedded database (for `duckdb-backend` feature)
- `tokio-postgres`: PostgreSQL client (for `postgres-backend` feature)
- `sqlparser`: SQL parsing
- `yaml-rust`: YAML processing
## Usage Examples
### File System Backend (Native Apps)
```rust
use data_modelling_sdk::storage::filesystem::FileSystemStorageBackend;
use data_modelling_sdk::model::ModelLoader;
let storage = FileSystemStorageBackend::new("/path/to/workspace");
let loader = ModelLoader::new(storage);
let result = loader.load_model("workspace_path").await?;
```
### Browser Storage Backend (WASM Apps)
```rust
use data_modelling_sdk::storage::browser::BrowserStorageBackend;
use data_modelling_sdk::model::ModelLoader;
let storage = BrowserStorageBackend::new("db_name", "store_name");
let loader = ModelLoader::new(storage);
let result = loader.load_model("workspace_path").await?;
```
### API Backend (Online Mode)
```rust
use data_modelling_sdk::storage::api::ApiStorageBackend;
use data_modelling_sdk::model::ModelLoader;
let storage = ApiStorageBackend::new("http://localhost:8081/api/v1", Some("session_id"));
let loader = ModelLoader::new(storage);
let result = loader.load_model("workspace_path").await?;
```
### Import/Export
```rust
use data_modelling_sdk::import::ODCSImporter;
use data_modelling_sdk::export::ODCSExporter;
// Import
let mut importer = ODCSImporter::new();
let result = importer.import(yaml_content)?;
// Export
let exporter = ODCSExporter::new();
let yaml = exporter.export_table(&table)?;
```
### Validation
```rust
use data_modelling_sdk::validation::{TableValidator, RelationshipValidator};
let table_validator = TableValidator::new();
let result = table_validator.detect_naming_conflicts(&tables)?;
let rel_validator = RelationshipValidator::new();
let (has_cycle, cycle_path) = rel_validator.check_circular_dependency(&relationships, source_id, target_id)?;
```
### Database Operations
```rust
use data_modelling_sdk::database::{DuckDBBackend, DatabaseBackend, SyncEngine};
// Initialize database
let backend = DuckDBBackend::new(".data-model.duckdb")?;
backend.initialize().await?;
// Sync YAML files to database
let sync_engine = SyncEngine::new(backend.clone());
let result = sync_engine.sync_workspace(workspace_path).await?;
// Execute SQL query
let result = backend.execute_query("SELECT * FROM tables").await?;
```
### CLI Database Commands
```bash
# Initialize database for a workspace
data-modelling-cli db init --workspace ./my-workspace --backend duckdb
# Sync YAML files to database
data-modelling-cli db sync --workspace ./my-workspace
# Check database status
data-modelling-cli db status --workspace ./my-workspace
# Export database to YAML files
data-modelling-cli db export --workspace ./my-workspace
# Query the database directly
data-modelling-cli query "SELECT * FROM tables" --workspace ./my-workspace --format json
```
## Testing
The SDK includes comprehensive tests:
- **Unit tests**: Individual module tests
- **Integration tests**: End-to-end workflows
- **Doctests**: Documentation examples
Run tests:
```bash
# All tests
cargo test --all-features
# Specific test suite
cargo test --test model_tests --features native-fs
# With output
cargo test --all-features -- --nocapture
```
## Development Workflow
### Pre-commit Hooks
The project uses pre-commit hooks for code quality:
```bash
# Install pre-commit
pip install pre-commit
# Install hooks
pre-commit install
# Run manually
pre-commit run --all-files
```
Hooks check:
- Rust formatting (`cargo fmt`)
- Rust linting (`cargo clippy`)
- Security audit (`cargo audit`)
- File formatting (trailing whitespace, end of file)
- YAML/TOML/JSON syntax
### CI/CD
GitHub Actions workflows:
- **Lint**: Format check, clippy, security audit
- **Test**: Unit and integration tests on Linux, macOS, Windows
- **Build**: Release build verification
- **Publish**: Automatic publishing to crates.io (manual trigger)
### Building
```bash
# Development build
cargo build
# Release build
cargo build --release
# With specific features
cargo build --features native-fs,git
# All features
cargo build --all-features
```
## File Structure
The SDK organizes files using a flat file naming convention:
```
base_directory/
├── .git/ # Git folder (if present)
│ └── hooks/ # Git hooks directory
│ ├── pre-commit # Exports database to YAML before commit
│ └── post-checkout # Syncs YAML to database after checkout
├── .data-model.toml # Database configuration file
├── .data-model.duckdb # DuckDB database file (if using DuckDB backend)
├── README.md # Repository files
├── workspace.yaml # Workspace definition with domains, systems, assets, and relationships
├── schemas/ # Schema reference directory
│ ├── README.md
│ ├── workspace-schema.json # Workspace schema with relationship definitions
│ ├── odcs-json-schema-v3.1.0.json
│ ├── odcl-json-schema-1.2.1.json
│ ├── odps-json-schema-latest.json
│ └── cads.schema.json
├── {workspace}_{domain}_{system}_{resource}.odcs.yaml # ODCS table files
├── {workspace}_{domain}_{system}_{resource}.odps.yaml # ODPS product files
├── {workspace}_{domain}_{system}_{resource}.cads.yaml # CADS asset files
├── {workspace}_{domain}_{system}_{resource}.bpmn.xml # BPMN process models
├── {workspace}_{domain}_{system}_{resource}.dmn.xml # DMN decision models
├── {workspace}_{domain}_{system}_{resource}.openapi.yaml # OpenAPI specifications
├── decisions/ # Architecture Decision Records
│ ├── index.yaml # Decision index with metadata
│ ├── 0001-decision-title.yaml # Individual decision files
│ └── ...
├── decisions-md/ # Markdown exports (auto-generated)
│ ├── 0001-decision-title.md
│ └── ...
├── knowledge/ # Knowledge Base articles
│ ├── index.yaml # Knowledge index with metadata
│ ├── 0001-article-title.yaml # Individual article files
│ └── ...
└── knowledge-md/ # Markdown exports (auto-generated)
├── 0001-article-title.md
└── ...
```
### Flat File Naming Convention
Files use the format: `{workspace}_{domain}_{system}_{resource}.{type}.yaml`
- **workspace**: The workspace name (e.g., `acme`)
- **domain**: The business domain (e.g., `sales`, `finance`)
- **system**: The system or service (e.g., `orders`, `payments`)
- **resource**: The specific resource name (e.g., `customers`, `transactions`)
- **type**: The asset type (e.g., `odcs`, `odps`, `cads`, `bpmn`, `dmn`, `openapi`)
### Workspace Configuration (`workspace.yaml`)
The workspace.yaml file is the central configuration containing:
- Workspace metadata (id, name, owner, description)
- Domain references
- System definitions
- Asset references (ODCS tables, ODPS products, CADS assets)
- **Relationships** - relationship definitions are now part of the workspace (no separate relationships.yaml)
### Database Configuration (`.data-model.toml`)
```toml
[database]
backend = "duckdb" # Backend type: "duckdb" or "postgres"
path = ".data-model.duckdb" # DuckDB file path (for DuckDB backend)
[postgres]
connection_string = "postgresql://user:pass@localhost/db" # PostgreSQL connection
pool_size = 5 # Connection pool size
[sync]
auto_sync = true # Enable automatic sync on file changes
watch = false # Enable file watching
[git]
hooks_enabled = true # Install Git hooks on db init
```
### Git Hooks
When database is initialized in a Git repository with `hooks_enabled = true`:
- **Pre-commit hook**: Exports database changes to YAML files before commit
- **Post-checkout hook**: Syncs YAML files to database after checkout/switch
Domain directories may also contain:
- `*.openapi.yaml` / `*.openapi.json`: OpenAPI specification files (can be referenced by CADS assets)
- `*.bpmn.xml`: BPMN 2.0 process model files (can be referenced by CADS assets)
- `*.dmn.xml`: DMN 1.3 decision model files (can be referenced by CADS assets)
## Key Design Decisions
1. **Storage Abstraction**: Uses trait-based storage backends to support multiple environments (native, WASM, API)
2. **Feature Flags**: Optional functionality gated behind features to minimize dependencies
3. **Async/Await**: Uses async traits for storage operations to support both native and WASM
4. **Error Handling**: Uses `anyhow::Result` for convenience, `thiserror` for structured errors
5. **UUID Strategy**: Uses UUIDv5 (deterministic) for model/table IDs to avoid random number generation requirements
6. **ODCS Primary Format**: ODCS v3.1.0 is the primary format, with legacy ODCL support for backward compatibility
7. **Domain-Based Organization**: Files are organized by business domain, with each domain containing its definition and associated ODCS/ODPS/CADS files
8. **Schema Reference**: JSON Schema definitions for all supported formats are maintained in `schemas/` directory for validation and reference
9. **camelCase Serialization**: All models use `#[serde(rename_all = "camelCase")]` for consistent JSON/YAML serialization matching ODCS conventions
10. **Crow's Feet Notation**: Relationships support standard crow's feet notation with EndpointCardinality (zeroOrOne, exactlyOne, zeroOrMany, oneOrMany) and FlowDirection (sourceToTarget, targetToSource, bidirectional)
## Common Patterns
### Error Handling
The SDK uses `anyhow::Result` for most operations:
```rust
use anyhow::Result;
pub async fn load_model(&self, path: &str) -> Result<ModelLoadResult, StorageError> {
// ...
}
```
### Storage Operations
All storage operations are async and use the `StorageBackend` trait:
```rust
#[async_trait(?Send)]
pub trait StorageBackend: Send + Sync {
async fn read_file(&self, path: &str) -> Result<Vec<u8>, StorageError>;
async fn write_file(&self, path: &str, content: &[u8]) -> Result<(), StorageError>;
// ...
}
```
### Import/Export Pattern
Importers convert external formats to SDK `Table` models, exporters convert SDK models to external formats:
```rust
pub trait Importer {
fn import(&mut self, content: &str) -> Result<ImportResult, ImportError>;
}
pub trait Exporter {
fn export_table(&self, table: &Table) -> Result<String, ExportError>;
}
```
## Schema Reference
The SDK maintains JSON Schema definitions for all supported formats in the `schemas/` directory:
- **ODCS v3.1.0**: `schemas/odcs-json-schema-v3.1.0.json` - Primary format for data contracts
- **ODCL v1.2.1**: `schemas/odcl-json-schema-1.2.1.json` - Legacy data contract format
- **ODPS**: `schemas/odps-json-schema-latest.json` - Data products linking to ODCS tables
- **CADS v1.0**: `schemas/cads.schema.json` - Compute assets (AI/ML models, applications, pipelines)
- **BPMN 2.0**: `schemas/bpmn-2.0.xsd` - Business Process Model and Notation (process models in native XML)
- **DMN 1.3**: `schemas/dmn-1.3.xsd` - Decision Model and Notation (decision models in native XML)
- **OpenAPI 3.1.1**: `schemas/openapi-3.1.1.json` - API specifications (stored in native YAML or JSON)
These schemas serve as authoritative references for:
- **Validation**: Validating imported YAML/JSON files against official specifications
- **Documentation**: Understanding the structure and fields of each format
- **Compliance**: Ensuring the SDK maintains full coverage of each specification
- **Reference**: Quick lookup for field definitions and types
See `schemas/README.md` for detailed information about each schema.
## Security
- Security advisories are tracked via `cargo audit`
- Configuration in `cargo-audit.toml` allows specific unmaintained warnings
- Pre-commit hooks run security audits automatically
## License
MIT License - See LICENSE file for details.