Crate schema_registry_lineage 

Schema Registry Lineage Tracking

This crate provides comprehensive lineage tracking capabilities for the LLM Schema Registry. It tracks dependencies between schemas, applications, data pipelines, and LLM models, enabling impact analysis and change propagation understanding.

Features

• Dependency Graph: Track schema-to-schema, schema-to-application, and schema-to-pipeline dependencies
• Transitive Dependencies: Calculate all transitive dependencies with depth control
• Impact Analysis: Analyze the impact of schema changes on downstream consumers
• Circular Dependency Detection: Detect and report circular dependencies
• Graph Algorithms: BFS, DFS, shortest path, topological sort
• Export Formats: GraphML, DOT (Graphviz), and JSON for visualization
• Thread-Safe: Concurrent access with Arc and RwLock

Quick Start

use schema_registry_lineage::{LineageEngine, LineageTracker, RelationType, SchemaChange};
use uuid::Uuid;

// Create a new lineage engine
let engine = LineageEngine::new();

// Track dependencies
let schema_a = Uuid::new_v4();
let schema_b = Uuid::new_v4();

engine.track_dependency(schema_a, schema_b, RelationType::DependsOn).await?;

// Get upstream dependencies
let upstream = engine.get_upstream(schema_a).await?;
println!("Schema A depends on {} schemas", upstream.len());

// Perform impact analysis
let change = SchemaChange::FieldRemoved {
    name: "email".to_string(),
};

let impact = engine.impact_analysis(schema_b, change).await?;
println!("Risk level: {:?}", impact.risk_level);
println!("Affected schemas: {}", impact.affected_schemas.len());

// Detect circular dependencies
let circular = engine.detect_circular().await?;
if !circular.is_empty() {
    println!("WARNING: {} circular dependencies detected", circular.len());
}

// Export for visualization
let graphml = engine.export_graphml()?;
std::fs::write("/tmp/lineage.graphml", graphml)?;

Architecture

The lineage tracking system consists of several key components:

• GraphStore: In-memory graph storage using petgraph
• Algorithms: Graph algorithms (BFS, DFS, cycles, etc.)
• Tracker: Dependency tracking operations
• ImpactAnalyzer: Schema change impact analysis
• Exporter: Export to GraphML, DOT, and JSON formats
• LineageEngine: Main orchestrator that combines all components

Examples

Tracking Schema Dependencies

use schema_registry_lineage::{LineageEngine, SchemaNode, DependencyTarget, RelationType};
use schema_registry_core::versioning::SemanticVersion;
use uuid::Uuid;

let engine = LineageEngine::new();

let user_schema = SchemaNode::new(
    Uuid::new_v4(),
    SemanticVersion::new(1, 0, 0),
    "com.example.User".to_string(),
);

let address_schema = SchemaNode::new(
    Uuid::new_v4(),
    SemanticVersion::new(1, 0, 0),
    "com.example.Address".to_string(),
);

// User composes Address
engine.track_dependency(
    user_schema.clone(),
    DependencyTarget::Schema(address_schema),
    RelationType::Composes,
).await?;

Impact Analysis

use schema_registry_lineage::{LineageEngine, LineageTracker, SchemaChange};
use uuid::Uuid;

let engine = LineageEngine::new();
let schema_id = Uuid::new_v4();

let change = SchemaChange::FieldTypeChanged {
    name: "age".to_string(),
    old_type: "int32".to_string(),
    new_type: "int64".to_string(),
};

let impact = engine.impact_analysis(schema_id, change).await?;

println!("Impact Report:");
println!("  Risk Level: {:?}", impact.risk_level);
println!("  Affected Schemas: {}", impact.affected_schemas.len());
println!("  Affected Apps: {}", impact.affected_applications.len());
println!("  Migration Complexity: {:.2}", impact.migration_complexity);
println!("  Estimated Effort: {:.1} hours", impact.estimated_effort_hours);

for recommendation in &impact.recommendations {
    println!("  - {}", recommendation);
}

Re-exports

pub use engine::LineageEngine;

pub use engine::LineageTracker;

pub use error::LineageError;

pub use error::Result;

pub use export::JsonEdge;

pub use export::JsonGraph;

pub use export::JsonGraphMetadata;

pub use export::JsonNode;

pub use export::LineageExporter;

pub use graph_store::GraphStats;

pub use graph_store::GraphStore;

pub use impact::ImpactAnalyzer;

pub use impact::ImpactSummary;

pub use tracker::DependencyTracker;

pub use tracker::DependencyTrackerImpl;

pub use types::CircularDependency;

pub use types::Dependency;

pub use types::DependencyGraph;

pub use types::DependencyTarget;

pub use types::Dependent;

pub use types::EntityType;

pub use types::ExternalEntity;

pub use types::ImpactReport;

pub use types::LineageFilter;

pub use types::RelationType;

pub use types::RiskLevel;

pub use types::SchemaChange;

pub use types::SchemaId;

pub use types::SchemaNode;

Modules

algorithms

Graph algorithms for lineage analysis

engine

Main lineage tracking engine

error

Error types for schema lineage tracking

export

Export lineage data to various formats

graph_store

In-memory graph storage using petgraph

impact

Impact analysis for schema changes

tracker

Dependency tracking operations

types

Core type definitions for schema lineage tracking