Crate inferadb_common_storage

Expand description

Shared storage backend abstraction for InferaDB services.

Provides the StorageBackend trait and related types that form the foundation for all storage operations in InferaDB. Both the Engine and Control services use this abstraction, enabling a unified storage layer.

§Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Service Layer                            │
│         (Engine API handlers, Control API handlers)         │
├─────────────────────────────────────────────────────────────┤
│                   Repository Layer                          │
│  RelationshipRepository │ OrganizationRepository │ etc.     │
│         (Domain logic, serialization, indexing)             │
├─────────────────────────────────────────────────────────────┤
│                 inferadb-storage                            │
│              StorageBackend trait                           │
│  (get, set, delete, get_range, compare_and_set, transaction) │
├──────────────┬───────────────────────────────────────────────┤
│ MemoryBackend│            LedgerBackend                      │
│   (testing)  │          (production)                         │
└──────────────┴───────────────────────────────────────────────┘

§Quick Start

use inferadb_common_storage::{MemoryBackend, StorageBackend};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create an in-memory backend for testing
    let backend = MemoryBackend::new();

    // Store a value
    backend.set(b"user:123".to_vec(), b"Alice".to_vec()).await?;

    // Retrieve it
    let value = backend.get(b"user:123").await?;
    assert_eq!(value.map(|b| b.to_vec()), Some(b"Alice".to_vec()));

    // Use transactions for atomic operations
    let mut txn = backend.transaction().await?;
    txn.set(b"counter".to_vec(), b"1".to_vec());
    txn.set(b"updated".to_vec(), b"true".to_vec());
    txn.commit().await?;

    Ok(())
}

§Compare-and-Set (CAS)

StorageBackend::compare_and_set provides atomic conditional updates for optimistic concurrency control. Pass expected: None for insert-if-absent or expected: Some(bytes) for update-if-unchanged. See the compare_and_set method documentation for full semantics, TTL interaction, transaction behavior, and retry patterns.

§Available Backends

Backend	Use Case	Persistence
`MemoryBackend`	Testing, development	No
`LedgerBackend` (in `inferadb-storage-ledger`)	Production	Yes

§Implementing a Backend

To implement a new storage backend:

Implement the StorageBackend trait
Implement a corresponding Transaction type
Map backend-specific errors to StorageError

See the memory module source for a reference implementation.

§Error Handling

All operations return StorageResult<T>, which wraps potential StorageError variants. Backends should map their internal errors to these standardized error types.

§Feature Flags

testutil: Enables the testutil module with shared test helpers (key/value generators, backend factories, assertion macros). Enable this in [dev-dependencies] for integration tests.

Re-exports§

pub use backend::DynBackend;
pub use backend::StorageBackend;
pub use backend::StorageBackendExt;
pub use backend::StorageRange;
pub use backend::to_storage_range;
pub use batch::BatchConfig;
pub use batch::BatchFlushStats;
pub use batch::BatchOperation;
pub use batch::BatchResult;
pub use batch::BatchWriter;
pub use buffered::BufferedBackend;
pub use cached::CacheConfig;
pub use cached::CachedBackend;
pub use cas_retry::CasRetryConfig;
pub use cas_retry::DEFAULT_CAS_RETRY_BASE_DELAY;
pub use cas_retry::DEFAULT_MAX_CAS_RETRIES;
pub use cas_retry::with_cas_retry;
pub use error::BoxError;
pub use error::ConfigError;
pub use error::StorageError;
pub use error::StorageResult;
pub use error::TimeoutContext;
pub use health::HealthMetadata;
pub use health::HealthProbe;
pub use health::HealthStatus;
pub use instrumented::InstrumentedBackend;
pub use memory::MemoryBackend;
pub use metrics::DEFAULT_MAX_TRACKED_ORGANIZATIONS;
pub use metrics::LatencyPercentiles;
pub use metrics::Metrics;
pub use metrics::MetricsCollector;
pub use metrics::MetricsSnapshot;
pub use metrics::OrganizationOperationSnapshot;
pub use rate_limiter::DEFAULT_MAX_ORGANIZATION_BUCKETS;
pub use rate_limiter::OrganizationExtractor;
pub use rate_limiter::RateLimitConfig;
pub use rate_limiter::RateLimitMetricsSnapshot;
pub use rate_limiter::RateLimitedBackend;
pub use rate_limiter::TokenBucketLimiter;
pub use size_limits::DEFAULT_MAX_KEY_SIZE;
pub use size_limits::DEFAULT_MAX_VALUE_SIZE;
pub use size_limits::SizeLimits;
pub use size_limits::validate_key_size;
pub use size_limits::validate_sizes;
pub use transaction::Transaction;
pub use types::CertId;
pub use types::ClientId;
pub use types::KeyValue;

Modules§

auth: Authentication primitives: signing keys, key stores, and audit logging. Authentication types and storage for Ledger-based token validation.
backend: Core StorageBackend trait defining the key-value storage interface. Storage backend trait definition.
batch: Batched write operations with automatic transaction splitting. Batch write operations for storage backends
buffered: Write-buffering wrapper for atomic multi-key commits. Write-buffering wrapper for atomic multi-key commits.
cached: Read-through cache wrapper for storage backends. Read-through cache wrapper for storage backends.
cas_retry: CAS (compare-and-set) conflict retry logic with jittered backoff. Retry logic for CAS (compare-and-set) conflict resolution.
error: Error types, result aliases, and diagnostic helpers. Storage error types and result alias.
health: Health check probes and status reporting. Health check types for storage backends.
instrumented: Instrumented wrapper that records per-operation timing and error metrics. Instrumented wrapper that records per-operation timing and error metrics.
memory: In-memory StorageBackend implementation for testing and development. In-memory storage backend implementation.
metrics: Operation metrics collection: counts, latencies, percentiles, and per-organization breakdowns. Storage metrics collection and monitoring
rate_limiter: Token-bucket rate limiter wrapper for storage backends. Rate limiting for storage backends.
size_limits: Key and value size limit validation. Key and value size validation for storage backends.
transaction: Transaction trait for atomic multi-operation commits. Transaction trait for atomic storage operations.
types: Common domain types: KeyValue, OrganizationSlug, VaultSlug, and other ID newtypes. Common types used across storage operations.

Structs§

OrganizationSlug: Snowflake-generated external identifier for an organization.
UserSlug: Snowflake-generated external identifier for a user.
VaultSlug: Snowflake-generated external identifier for a vault.
Zeroizing: Zeroizing is a a wrapper for any Z: Zeroize type which implements a Drop handler which zeroizes dropped values.