Crate rivven_schema

Source

Expand description

§Rivven Schema Registry

A high-performance, Confluent-compatible Schema Registry for the Rivven event streaming platform.

§Features

Confluent API Compatible: Drop-in replacement for Confluent Schema Registry
Multiple Schema Formats: Avro, JSON Schema, Protobuf
Schema Evolution: Forward, backward, full, and transitive compatibility checking
Schema Fingerprinting: MD5 and SHA-256 for deduplication
Storage Backends: In-memory (dev), Broker-backed (production)
External Registries: Confluent Schema Registry client, AWS Glue integration
High Performance: In-memory caching with async operations
Production Ready: K8s health checks, graceful shutdown

§Architecture

The Schema Registry is responsible for:

Storing schema definitions (as text strings)
Versioning schemas with subjects
Checking compatibility between schema versions
Serving schemas via REST API

The registry does NOT encode/decode data — that’s the job of producers and consumers. Use rivven-connect for codecs (Avro, Protobuf, JSON).

┌─────────────────────────────────────────────────────────────┐
│                    Rivven Schema Registry                    │
├─────────────────────────────────────────────────────────────┤
│  REST API (Confluent-compatible)                            │
│  ├── POST /subjects/{subject}/versions                      │
│  ├── GET  /schemas/ids/{id}                                 │
│  ├── GET  /subjects/{subject}/versions/{version}            │
│  ├── POST /compatibility/subjects/{subject}/versions/{ver}  │
│  ├── GET  /config                                           │
│  └── GET  /health, /health/live, /health/ready              │
├─────────────────────────────────────────────────────────────┤
│  Schema Management Layer                                     │
│  ├── Schema Parsing (syntax validation)                     │
│  ├── Compatibility Checking (all Confluent modes)           │
│  ├── Fingerprint/ID Generation (MD5, SHA-256)               │
│  └── Caching with deduplication                              │
├─────────────────────────────────────────────────────────────┤
│  Storage Backends                                            │
│  ├── Memory (development/testing)                           │
│  ├── Broker (production - durable, replicated)              │
│  └── External (Confluent, AWS Glue)                         │
└─────────────────────────────────────────────────────────────┘

§Quick Start

§As a Library

use rivven_schema::{SchemaRegistry, RegistryConfig, SchemaType};

// Create an in-memory registry
let config = RegistryConfig::memory();
let registry = SchemaRegistry::new(config).await?;

// Register a schema
let avro_schema = r#"{"type": "record", "name": "User", "fields": [{"name": "id", "type": "long"}]}"#;
let schema_id = registry.register("user-value", SchemaType::Avro, avro_schema).await?;

// Retrieve the schema
let schema = registry.get_by_id(schema_id).await?;
println!("Schema: {}", schema.schema);

§As a Standalone Server

# Start with in-memory storage (development/testing)
rivven-schema serve --port 8081

For production, use broker-backed storage (requires broker feature):

use rivven_schema::{RegistryConfig, BrokerStorageConfig};

let config = RegistryConfig::broker(
    BrokerStorageConfig::new("localhost:9092")
        .with_topic("_schemas")
        .with_replication_factor(3)
);

§Compatibility Modes

Mode	Description
`BACKWARD`	New schema can read old data (default)
`BACKWARD_TRANSITIVE`	New schema can read all previous versions
`FORWARD`	Old schema can read new data
`FORWARD_TRANSITIVE`	All previous schemas can read new data
`FULL`	Both backward and forward compatible
`FULL_TRANSITIVE`	Full compatibility with all versions
`NONE`	No compatibility checking

§Confluent Wire Format

When using Avro with Schema Registry, data is encoded with a 5-byte header:

[0x00][schema_id: 4 bytes big-endian][avro_binary_data]

This is compatible with Kafka producers/consumers using Confluent serializers.

§Best Practices

Use Avro in Production: Schema evolution with compatibility checking
Subject Naming: Use {topic}-key and {topic}-value convention
Compatibility Level: Start with BACKWARD for safe evolution
Versioning: Never delete schemas, only deprecate

§Note on Codecs

The Schema Registry does not include codecs for encoding/decoding data. That’s the responsibility of producers and consumers (connectors). Use rivven-connect for Avro/Protobuf/JSON codecs with Confluent wire format.

Re-exports§

pub use auth::AuthConfig;
pub use auth::AuthState;
pub use auth::SchemaPermission;
pub use auth::ServerAuthState;
pub use compatibility::CompatibilityChecker;
pub use compatibility::CompatibilityLevel;
pub use compatibility::CompatibilityResult;
pub use config::RegistryConfig;
pub use config::StorageConfig;
pub use error::error_codes;
pub use error::SchemaError;
pub use error::SchemaResult;
pub use fingerprint::SchemaFingerprint;
pub use registry::SchemaRegistry;
pub use server::RegistryMode;
pub use server::SchemaServer;
pub use server::ServerConfig;
pub use server::ServerState;
pub use storage::MemoryStorage;
pub use storage::Storage;
pub use storage::StorageBackend;
pub use types::Schema;
pub use types::SchemaContext;
pub use types::SchemaId;
pub use types::SchemaMetadata;
pub use types::SchemaReference;
pub use types::SchemaVersion;
pub use types::Subject;
pub use types::SubjectVersion;
pub use types::ValidationLevel;
pub use types::ValidationReport;
pub use types::ValidationResult;
pub use types::ValidationRule;
pub use types::ValidationRuleType;
pub use types::ValidationSummary;
pub use types::VersionState;
pub use validation::ValidationEngine;
pub use validation::ValidationEngineConfig;

Modules§

auth: Authentication & Authorization middleware for Schema Registry
compatibility: Schema compatibility checking
config: Schema Registry configuration
error: Schema Registry errors
fingerprint: Schema fingerprinting for deduplication
registry: Schema Registry - main interface
server: HTTP Server for Schema Registry
storage: Storage backends for Schema Registry
types: Schema types and data structures
validation: Content Validation Engine

Enums§

SchemaType: Schema type (format) for schema registry