Crate ipfrs_storage 

IPFRS Storage - Block storage and retrieval

This crate provides the storage layer for IPFRS including:

• Sled, ParityDB, and in-memory block stores
• LRU and tiered caching
• Bloom filter for fast existence checks
• Streaming interface for large blocks
• Content-defined chunking and deduplication
• Transparent block compression (Zstd, Lz4, Snappy)
• Pin management for preventing GC
• Hot/cold tiering with access tracking
• Garbage collection (mark-and-sweep)
• CAR format export/import for backups
• Encryption at rest (ChaCha20-Poly1305, AES-256-GCM)
• Version Control System for differentiable storage (Git for Tensors)
• RAFT consensus protocol for distributed storage
• Network transport abstraction (TCP, QUIC with TLS) for multi-node RAFT clusters
• Cluster coordinator with automatic failover and re-election
• Multi-datacenter support with latency-aware routing and cross-datacenter replication
• Eventual consistency with version vectors and conflict resolution
• GraphQL query interface for flexible metadata querying
• ARM profiler with NEON SIMD optimization and low-power tuning
• Production-grade metrics and observability for monitoring (Prometheus, OpenTelemetry)
• Circuit breaker pattern for fault-tolerant external service calls
• Unified health check system for liveness and readiness monitoring
• TTL support for automatic block expiration
• Advanced retry logic with exponential backoff and jitter
• Optimized S3 multipart uploads for large blocks
• Rate limiting for controlling request rates to backends
• Write coalescing for batching similar writes
• Workload simulation for testing and benchmarking
• Automatic configuration tuning based on workload patterns
• Comprehensive profiling system with comparative analysis and regression detection
• Storage pool manager for multi-backend routing with intelligent load balancing
• Quota management system for per-tenant storage limits and bandwidth control
• Lifecycle policies for automatic data tiering, archival, and expiration
• Predictive prefetching with access pattern learning and adaptive depth control
• Cost analytics and optimization for cloud storage (AWS, Azure, GCP)
• Safe, transactional schema migration framework with rollback support

Re-exports

pub use block_garbage_collector::BgcBlockRecord;

pub use block_garbage_collector::BgcCollectorConfig;

pub use block_garbage_collector::BgcCollectorStats;

pub use block_garbage_collector::BgcGcPolicy;

pub use block_garbage_collector::BgcGcResult;

pub use block_garbage_collector::BgcSweepResult;

pub use block_garbage_collector::BlockGarbageCollector;

pub use content_dedup_index::ContentDedupConfig;

pub use content_dedup_index::ContentDedupResult;

pub use content_dedup_index::ContentDedupStats;

pub use content_dedup_index::ContentDeduplicationIndex;

pub use content_dedup_index::ContentHash;

pub use content_dedup_index::DedupIndexError;

pub use block_validator::StorageBlockValidator;

pub use block_validator::ValidationReport;

pub use block_validator::ValidationResult;

pub use block_validator::ValidationRule;

pub use block_validator::ValidatorStats;

pub use block_verifier::BlockRecord;

pub use block_verifier::StorageBlockVerifier;

pub use block_verifier::VerificationReport;

pub use block_verifier::VerificationResult;

pub use block_verifier::VerifierStats;

pub use index_recovery::IndexEntry as IrIndexEntry;

pub use index_recovery::IndexRecovery;

pub use index_recovery::RawBlock;

pub use index_recovery::RecoveryConfig;

pub use index_recovery::RecoveryStats;

pub use index_recovery::RecoveryStatus;

pub use storage_metrics_collector::AggregatedStats;

pub use storage_metrics_collector::CollectorConfig;

pub use storage_metrics_collector::CollectorStats;

pub use storage_metrics_collector::MetricKind;

pub use storage_metrics_collector::MetricSample;

pub use storage_metrics_collector::MetricSeries;

pub use storage_metrics_collector::StorageMetricsCollector as SmcStorageMetricsCollector;

pub use storage_metrics_collector::TimeBucket;

pub use eviction_policy::CacheEntry as EvictionCacheEntry;

pub use eviction_policy::EvictionCandidate;

pub use eviction_policy::EvictionStrategy;

pub use eviction_policy::PolicyStats as EvictionPolicyStats;

pub use eviction_policy::StorageEvictionPolicy;

pub use audit_trail::AuditEntry;

pub use audit_trail::AuditEventType;

pub use audit_trail::AuditFilter;

pub use audit_trail::AuditTrail;

pub use audit_trail::AuditTrailConfig;

pub use audit_trail::AuditTrailStats;

pub use block_migration_planner::BlockMigrationPlanner;

pub use block_migration_planner::BmpBlockMeta;

pub use block_migration_planner::BmpMigrationPlan;

pub use block_migration_planner::BmpPlanStatus;

pub use block_migration_planner::BmpPlannerConfig;

pub use block_migration_planner::BmpPlannerStats;

pub use block_migration_planner::BmpPriorityPolicy;

pub use compression_pipeline::CompressionAlgo as CpCompressionAlgo;

pub use compression_pipeline::CompressionError as CpCompressionError;

pub use compression_pipeline::CompressionHint as CpCompressionHint;

pub use compression_pipeline::CompressionResult as CpCompressionResult;

pub use compression_pipeline::PipelineConfig as CpPipelineConfig;

pub use compression_pipeline::PipelineStage as CpPipelineStage;

pub use compression_pipeline::PipelineStats as CpPipelineStats;

pub use compression_pipeline::StorageCompressionPipeline;

pub use compression_registry::CodecProfile;

pub use compression_registry::CodecRecommendation;

pub use compression_registry::CompressionCodec as RegistryCompressionCodec;

pub use compression_registry::CompressionRegistryStats;

pub use compression_registry::DataCharacteristics;

pub use compression_registry::StorageCompressionRegistry;

pub use compactor::CompactionResult as RegionCompactionResult;

pub use compactor::CompactionState as RegionCompactionState;

pub use compactor::CompactorConfig as RegionCompactorConfig;

pub use compactor::CompactorStats as RegionCompactorStats;

pub use compactor::FragmentationReport as RegionFragmentationReport;

pub use compactor::StorageCompactor;

pub use cold_storage::fnv1a_cold;

pub use cold_storage::ColdStorageManager;

pub use cold_storage::ColdStorageStats;

pub use cold_storage::StorageTier as CsStorageTier;

pub use cold_storage::TierPolicy as CsTierPolicy;

pub use cold_storage::TieredBlock as CsTieredBlock;

pub use corruption_repair::BlockWithParity;

pub use corruption_repair::CorruptionRepairer;

pub use corruption_repair::CorruptionReport;

pub use corruption_repair::CorruptionType;

pub use corruption_repair::RepairAction;

pub use corruption_repair::RepairConfig;

pub use corruption_repair::RepairStats;

pub use data_integrity_auditor::compute_adler32 as dia_compute_adler32;

pub use data_integrity_auditor::compute_checksum as dia_compute_checksum;

pub use data_integrity_auditor::compute_crc32 as dia_compute_crc32;

pub use data_integrity_auditor::compute_fnv_xor64 as dia_compute_fnv_xor64;

pub use data_integrity_auditor::AuditConfig;

pub use data_integrity_auditor::AuditResult;

pub use data_integrity_auditor::AuditorStats;

pub use data_integrity_auditor::BlockChecksum;

pub use data_integrity_auditor::ChecksumAlgo;

pub use data_integrity_auditor::DataIntegrityAuditor;

pub use data_integrity_auditor::RepairRecord;

pub use data_integrity_checker::BlockRecord as DicBlockRecord;

pub use data_integrity_checker::DataIntegrityChecker;

pub use data_integrity_checker::IntegrityCheckerConfig;

pub use data_integrity_checker::IntegrityReport;

pub use data_integrity_checker::IntegrityStats;

pub use data_integrity_checker::IntegrityStatus;

pub use deduplication_pipeline::BlockEntry;

pub use deduplication_pipeline::DedupPipelineStats;

pub use deduplication_pipeline::DedupResult as DpDedupResult;

pub use deduplication_pipeline::DedupStage;

pub use deduplication_pipeline::DeduplicationPipeline;

pub use deduplication_pipeline::PipelineConfig;

pub use garbage_collector::GcError;

pub use garbage_collector::GcObject;

pub use garbage_collector::GcObjectId;

pub use garbage_collector::GcPhase;

pub use garbage_collector::StorageGarbageCollector;

pub use garbage_collector::StorageGcConfig;

pub use garbage_collector::StorageGcRun;

pub use garbage_collector::StorageGcStats;

pub use quota_enforcer::NamespaceQuota as EnforcerNamespaceQuota;

pub use quota_enforcer::QuotaEnforcerConfig;

pub use quota_enforcer::QuotaEnforcerStats;

pub use quota_enforcer::QuotaLevel;

pub use quota_enforcer::StorageQuotaEnforcer;

pub use snapshot_diff::DiffEntry;

pub use snapshot_diff::DiffKind;

pub use snapshot_diff::DiffStats as SnapshotDiffStats;

pub use snapshot_diff::SnapshotDiffResult;

pub use snapshot_diff::SnapshotEntry as SnapshotDiffEntry;

pub use snapshot_diff::StorageSnapshotDiff;

pub use snapshot_manager::fnv1a_64 as snapshot_fnv1a_64;

pub use snapshot_manager::LegacySnapshot;

pub use snapshot_manager::LegacySnapshotDiff as ManagerSnapshotDiff;

pub use snapshot_manager::LegacySnapshotEntry;

pub use snapshot_manager::LegacyStorageSnapshotManager;

pub use snapshot_manager::SnapshotConfig;

pub use snapshot_manager::SnapshotDelta;

pub use snapshot_manager::SnapshotEntry as SsmSnapshotEntry;

pub use snapshot_manager::SnapshotError;

pub use snapshot_manager::SnapshotId;

pub use snapshot_manager::SnapshotKind;

pub use snapshot_manager::SnapshotManagerStats;

pub use snapshot_manager::SnapshotState;

pub use snapshot_manager::SnapshotStats as SsmSnapshotStats;

pub use snapshot_manager::SsmSnapshot;

pub use snapshot_manager::StorageSnapshot;

pub use snapshot_manager::StorageSnapshotManager;

pub use snapshot_manager::StorageState as SsmStorageState;

pub use tier_migration_engine::BlockMeta as TmBlockMeta;

pub use tier_migration_engine::MigrationAction as TmMigrationAction;

pub use tier_migration_engine::MigrationReason as TmMigrationReason;

pub use tier_migration_engine::MigrationResult as TmMigrationResult;

pub use tier_migration_engine::MigratorConfig;

pub use tier_migration_engine::MigratorStats;

pub use tier_migration_engine::StorageTier as TmStorageTier;

pub use tier_migration_engine::StorageTierMigrator;

pub use tier_migration_engine::TierPolicy as TmTierPolicy;

pub use content_addressable_cache::CacheConfig as CacCacheConfig;

pub use content_addressable_cache::CacheEntry as CacCacheEntry;

pub use content_addressable_cache::CacheError;

pub use content_addressable_cache::CacheStats as CacCacheStats;

pub use content_addressable_cache::ContentAddressableCache;

pub use content_addressable_cache::EvictionPolicy as CacEvictionPolicy;

pub use content_addressable_cache::LruNode;

pub use content_addressed_cache_v2::Cac2CacheConfig;

pub use content_addressed_cache_v2::Cac2CacheStats;

pub use content_addressed_cache_v2::Cac2Cid;

pub use content_addressed_cache_v2::Cac2Entry;

pub use content_addressed_cache_v2::Cac2EvictionRecord;

pub use content_addressed_cache_v2::Cac2Tier;

pub use content_addressed_cache_v2::ContentAddressedCacheV2;

pub use access_predictor::AccessEvent;

pub use access_predictor::AccessPattern as PredictorAccessPattern;

pub use access_predictor::PredictionResult;

pub use access_predictor::PredictorStats;

pub use access_predictor::StorageAccessPredictor;

pub use analyzer::Category;

pub use analyzer::Difficulty;

pub use analyzer::OperationStats;

pub use analyzer::OptimizationRecommendation;

pub use analyzer::Priority;

pub use analyzer::SizeDistribution;

pub use analyzer::StorageAnalysis;

pub use analyzer::StorageAnalyzer;

pub use analyzer::WorkloadCharacterization;

pub use analyzer::WorkloadType;

pub use arm_profiler::hash_block;

pub use arm_profiler::ArmFeatures;

pub use arm_profiler::ArmPerfCounter;

pub use arm_profiler::ArmPerfReport;

pub use arm_profiler::LowPowerBatcher;

pub use arm_profiler::PowerProfile;

pub use arm_profiler::PowerStats;

pub use auto_tuner::AutoTuner;

pub use auto_tuner::AutoTunerConfig;

pub use auto_tuner::TuningPresets;

pub use auto_tuner::TuningRecommendation;

pub use auto_tuner::TuningReport;

pub use batch::batch_delete;

pub use batch::batch_get;

pub use batch::batch_has;

pub use batch::batch_put;

pub use batch::BatchConfig;

pub use batch::BatchResult;

pub use block_index::BlockIndexEntry;

pub use block_index::BlockIndexStats;

pub use block_index::IndexEntry;

pub use block_index::IndexKey;

pub use block_index::IndexQuery;

pub use block_index::IndexStats;

pub use block_index::SecondaryBlockIndex;

pub use block_index::StorageBlockIndex;

pub use block_manifest::fnv1a;

pub use block_manifest::ManifestEntry;

pub use block_manifest::ManifestFilter;

pub use block_manifest::ManifestStats;

pub use block_manifest::StorageBlockManifest;

pub use block_packer::fnv1a as packer_fnv1a;

pub use block_packer::Pack;

pub use block_packer::PackEntry;

pub use block_packer::PackerConfig;

pub use block_packer::PackerStats;

pub use block_packer::StorageBlockPacker;

pub use block_stream::BlockChunk;

pub use block_stream::BlockStreamIterator;

pub use block_stream::BlockStreamState;

pub use block_stream::StreamConfig as BlockStreamConfig;

pub use block_stream::StreamStats as BlockStreamStats;

pub use blockstore::SledBlockStore;

pub use blockstore::BlockStoreConfig;

pub use blockstore::DeduplicationStats;

pub use blockstore::DeduplicationStatsSnapshot;

pub use bloom::BloomBlockStore;

pub use bloom::BloomConfig;

pub use bloom::BloomFilter;

pub use bloom::BloomFilterConfig;

pub use bloom::BloomSnapshot;

pub use bloom::BloomStats;

pub use bloom::CidBloomFilter;

pub use cache::BlockCache;

pub use cache::CacheConfig;

pub use cache::CacheStats;

pub use cache::CacheStatsSnapshot;

pub use cache::CachedBlockStore;

pub use cache::LegacyCacheStats;

pub use cache::TieredBlockCache;

pub use cache::TieredCacheStats;

pub use cache::TieredCachedBlockStore;

pub use car::export_to_car;

pub use car::import_from_car;

pub use car::CarHeader;

pub use car::CarReadStats;

pub use car::CarReader;

pub use car::CarWriteStats;

pub use car::CarWriter;

pub use circuit_breaker::CircuitBreaker;

pub use circuit_breaker::CircuitState;

pub use circuit_breaker::CircuitStats;

pub use cluster::ClusterConfig;

pub use cluster::ClusterCoordinator;

pub use cluster::ClusterStats;

pub use cluster::NodeHealth;

pub use cluster::NodeInfo;

pub use coalesce::CoalesceConfig;

pub use coalesce::CoalesceStats;

pub use coalesce::CoalescingBlockStore;

pub use compaction::CompactionConfig;

pub use compaction::CompactionScheduler;

pub use cost_analytics::CloudProvider;

pub use cost_analytics::CostAnalyzer;

pub use cost_analytics::CostBreakdown;

pub use cost_analytics::CostProjection;

pub use cost_analytics::CostTier;

pub use cost_analytics::TierCostModel;

pub use cost_analytics::TierOption;

pub use cost_analytics::TierRecommendation;

pub use datacenter::CrossDcStats;

pub use datacenter::Datacenter;

pub use datacenter::DatacenterId;

pub use datacenter::LatencyAwareSelector;

pub use datacenter::MultiDatacenterCoordinator;

pub use datacenter::Region;

pub use datacenter::ReplicationPolicy;

pub use dedup::ChunkingConfig;

pub use dedup::DedupBlockStore;

pub use dedup::DedupStats;

pub use diagnostics::BenchmarkComparison;

pub use diagnostics::DiagnosticsReport;

pub use diagnostics::HealthMetrics;

pub use diagnostics::PerformanceMetrics;

pub use diagnostics::StorageDiagnostics;

pub use eventual_consistency::ConflictResolution;

pub use eventual_consistency::ConsistencyLevel;

pub use eventual_consistency::EventualStore;

pub use eventual_consistency::EventualStoreStats;

pub use eventual_consistency::VersionVector;

pub use eventual_consistency::VersionedValue;

pub use exporters::BatchExporter;

pub use exporters::ExportFormat;

pub use exporters::MetricExporter;

pub use gc::SledSnapshotPinRegistry;

pub use gc::snapshot_pin_id;

pub use gc::GarbageCollector;

pub use gc::GcConfig;

pub use gc::GcPolicy;

pub use gc::GcResult;

pub use gc::GcScheduler;

pub use gc::GcStats;

pub use gc::GcStatsSnapshot;

pub use gc::OrphanGarbageCollector;

pub use gc::OrphanGcConfig;

pub use gc::OrphanGcResult;

pub use gc::SnapshotPinRegistry;

pub use gc_planner::GCCandidate;

pub use gc_planner::GCConfig;

pub use gc_planner::GCPlan;

pub use gc_planner::GCPlannerStats;

pub use gc_planner::StorageGCPlanner;

pub use gradient::CompressionStats;

pub use gradient::DeltaEncoder;

pub use gradient::GradientData;

pub use gradient::GradientStore;

pub use gradient::ProvenanceMetadata;

pub use health::AggregateHealthResult;

pub use health::DetailedHealthStatus;

pub use health::HealthCheck;

pub use health::HealthCheckResult;

pub use health::HealthChecker;

pub use health::HealthStatus;

pub use health::SimpleHealthCheck;

pub use health_monitor::HealthMonitorConfig;

pub use health_monitor::HealthMonitorStats;

pub use health_monitor::MonitorHealthCheck;

pub use health_monitor::MonitorHealthStatus;

pub use health_monitor::StorageHealthMonitor;

pub use helpers::blockchain_stack;

pub use helpers::cache_stack;

pub use helpers::cdn_edge_stack;

pub use helpers::coalescing_memory_stack;

pub use helpers::deduplicated_production_stack;

pub use helpers::development_stack;

pub use helpers::embedded_stack;

pub use helpers::ingestion_stack;

pub use helpers::iot_stack;

pub use helpers::media_streaming_stack;

pub use helpers::memory_stack;

pub use helpers::ml_model_stack;

pub use helpers::monitored_production_stack;

pub use helpers::production_stack;

pub use helpers::read_optimized_stack;

pub use helpers::resilient_stack;

pub use helpers::testing_stack;

pub use helpers::ttl_production_stack;

pub use helpers::write_optimized_stack;

pub use helpers::StorageStackBuilder;

pub use storage_health_monitor::ShmAlert;

pub use storage_health_monitor::ShmCategory;

pub use storage_health_monitor::ShmHealthSnapshot;

pub use storage_health_monitor::ShmMonitorConfig;

pub use storage_health_monitor::ShmMonitorStats;

pub use storage_health_monitor::ShmProbe;

pub use storage_health_monitor::ShmProbeId;

pub use storage_health_monitor::ShmSeverity;

pub use storage_health_monitor::ShmStatus;

pub use storage_health_monitor::StorageHealthMonitor as ShmStorageHealthMonitor;

pub use lifecycle::BlockMetadata as LifecycleBlockMetadata;

pub use lifecycle::LifecycleAction;

pub use lifecycle::LifecycleActionResult;

pub use lifecycle::LifecycleCondition;

pub use lifecycle::LifecyclePolicyConfig;

pub use lifecycle::LifecyclePolicyManager;

pub use lifecycle::LifecycleRule;

pub use lifecycle::LifecycleStatsSnapshot;

pub use lifecycle::StorageTier;

pub use memory::MemoryBlockStore;

pub use metrics::MetricsBlockStore;

pub use metrics::StorageMetrics;

pub use migration::estimate_migration;

pub use migration::migrate_storage;

pub use migration::migrate_storage_batched;

pub use migration::migrate_storage_verified;

pub use migration::migrate_storage_with_progress;

pub use migration::validate_migration;

pub use migration::BlockMigrationStats;

pub use migration::MigrationConfig;

pub use migration::MigrationEstimate;

pub use migration::StorageMigrator;

pub use migration::MigrationError;

pub use migration::MigrationPlan;

pub use migration::MigrationRecord;

pub use migration::MigrationRunner;

pub use migration::MigrationStats;

pub use migration::MigrationStatsSnapshot;

pub use migration::MigrationStatus;

pub use migration::MigrationStep;

pub use migration::SchemaVersion;

pub use otel::OtelBlockStore;

pub use pinning::PinInfo;

pub use pinning::PinManager;

pub use pinning::PinSet;

pub use pinning::PinStatsSnapshot;

pub use pinning::PinType;

pub use pool::BackendConfig;

pub use pool::BackendId;

pub use pool::PoolConfig;

pub use pool::PoolStats;

pub use pool::RoutingStrategy;

pub use pool::StoragePool;

pub use prefetch::AccessPattern as PrefetchAccessPattern;

pub use prefetch::PredictivePrefetcher;

pub use prefetch::PrefetchConfig;

pub use prefetch::PrefetchPrediction;

pub use prefetch::PrefetchStatsSnapshot;

pub use profiler::ComparativeProfiler;

pub use profiler::ComparisonReport;

pub use profiler::ProfileConfig;

pub use profiler::ProfileReport;

pub use profiler::RegressionDetector;

pub use profiler::RegressionResult;

pub use profiler::StorageProfiler;

pub use profiling::BatchProfiler;

pub use profiling::LatencyHistogram;

pub use profiling::PerformanceProfiler;

pub use profiling::ThroughputTracker;

pub use prometheus::PrometheusExporter;

pub use prometheus::PrometheusExporterBuilder;

pub use query_optimizer::OptimizerConfig;

pub use query_optimizer::QueryLogEntry;

pub use query_optimizer::QueryOptimizer;

pub use query_optimizer::QueryPlan;

pub use query_optimizer::QueryStrategy;

pub use query_optimizer::Recommendation;

pub use query_optimizer::RecommendationCategory;

pub use query_optimizer::RecommendationPriority;

pub use quota::QuotaBlockStore;

pub use quota::QuotaConfig;

pub use quota::QuotaManager;

pub use quota::QuotaManagerConfig;

pub use quota::QuotaReport;

pub use quota::QuotaStatus;

pub use quota::QuotaUsageSnapshot;

pub use quota::TenantId;

pub use quota::ViolationType;

pub use read_ahead::PrefetchHint;

pub use read_ahead::ReadAheadPattern;

pub use read_ahead::ReadAheadScheduler;

pub use read_ahead::ReadAheadStats;

pub use chunk_manager::fnv1a_u64 as chunk_fnv1a_u64;

pub use chunk_manager::Chunk;

pub use chunk_manager::ChunkManagerStats;

pub use chunk_manager::ChunkState;

pub use chunk_manager::ChunkedObject;

pub use chunk_manager::StorageChunkManager;

pub use fragmentation_analyzer::CompactionCandidate;

pub use fragmentation_analyzer::FragmentationReport;

pub use fragmentation_analyzer::StorageExtent;

pub use fragmentation_analyzer::StorageFragmentationAnalyzer;

pub use heatmap_tracker::HeatBucket;

pub use heatmap_tracker::HeatEntry;

pub use heatmap_tracker::HeatmapStats;

pub use heatmap_tracker::StorageHeatmapTracker;

pub use quota_registry::QuotaEntry;

pub use quota_registry::QuotaKind;

pub use quota_registry::QuotaViolation;

pub use quota_registry::RegistryStats;

pub use quota_registry::StorageQuotaRegistry;

pub use quota_registry::ViolationType as RegistryViolationType;

pub use raft::AppendEntriesRequest;

pub use raft::AppendEntriesResponse;

pub use raft::Command;

pub use raft::LogEntry;

pub use raft::LogIndex;

pub use raft::NodeId;

pub use raft::NodeState;

pub use raft::RaftConfig;

pub use raft::RaftNode;

pub use raft::RaftStats;

pub use raft::RequestVoteRequest;

pub use raft::RequestVoteResponse;

pub use raft::Term;

pub use rate_limit::RateLimitAlgorithm;

pub use rate_limit::RateLimitConfig;

pub use rate_limit::RateLimitStats;

pub use rate_limit::RateLimiter;

pub use replication::ConflictStrategy;

pub use replication::ReplicationManager;

pub use replication::ReplicationState;

pub use replication::Replicator;

pub use replication::SyncResult;

pub use replication::SyncStrategy;

pub use replication_tracker::BlockReplicationEntry;

pub use replication_tracker::ReplicaLocation;

pub use replication_tracker::ReplicationStats as BlockReplicationStats;

pub use replication_tracker::ReplicationStatus;

pub use replication_tracker::ReplicationTask;

pub use replication_tracker::StorageReplicationTracker;

pub use retention_policy::BlockRecord as RetentionBlockRecord;

pub use retention_policy::PolicyDecision;

pub use retention_policy::RetentionAction;

pub use retention_policy::RetentionEntry;

pub use retention_policy::RetentionPolicyStats;

pub use retention_policy::RetentionRule;

pub use retention_policy::RetentionStats;

pub use retention_policy::StorageRetentionPolicy;

pub use retention_policy::StorageRetentionPolicyEngine;

pub use retention_policy::TickRetentionAction;

pub use retention_policy::TickRetentionRule;

pub use retry::BackoffStrategy;

pub use retry::JitterType;

pub use retry::RetryPolicy;

pub use retry::RetryStats;

pub use retry::Retryable;

pub use safetensors::ChunkConfig;

pub use safetensors::ChunkedTensor;

pub use safetensors::DType;

pub use safetensors::ModelStats;

pub use safetensors::SafetensorsHeader;

pub use safetensors::SafetensorsManifest;

pub use safetensors::SafetensorsStore;

pub use safetensors::TensorInfo;

pub use streaming::BlockReader;

pub use streaming::ByteRange;

pub use streaming::PartialBlock;

pub use streaming::StreamConfig;

pub use streaming::StreamingBlockStore;

pub use streaming::StreamingWriter;

pub use tier_balancer::BalancerStats;

pub use tier_balancer::MoveTask;

pub use tier_balancer::StorageTierBalancer;

pub use tier_balancer::TierKind;

pub use tier_balancer::TierStatus;

pub use tier_manager::BlockTierRecord;

pub use tier_manager::StorageTier as TierManagerStorageTier;

pub use tier_manager::StorageTierManager;

pub use tier_manager::TierPolicy;

pub use tier_manager::TierStats;

pub use tier_manager::TierStatsSnapshot as TierManagerStatsSnapshot;

pub use tier_manager::TierTransition;

pub use tiering::AccessStats;

pub use tiering::AccessTracker;

pub use tiering::Tier;

pub use tiering::TierConfig;

pub use tiering::TierStatsSnapshot;

pub use tiering::TieredStore;

pub use traits::BlockStore as BlockStoreTrait;

pub use transport::InMemoryTransport;

pub use transport::Message as TransportMessage;

pub use transport::TcpTransport;

pub use transport::Transport;

pub use transport::TransportConfig;

pub use ttl::TtlBlockStore;

pub use ttl::TtlCleanupResult;

pub use ttl::TtlConfig;

pub use ttl::TtlStats;

pub use utils::compute_cid;

pub use utils::compute_total_size;

pub use utils::create_block;

pub use utils::create_blocks_batch;

pub use utils::deduplicate_blocks;

pub use utils::estimate_compression_ratio;

pub use utils::extract_cids;

pub use utils::filter_blocks_by_size;

pub use utils::find_duplicates;

pub use utils::generate_compressible_blocks;

pub use utils::generate_compressible_data;

pub use utils::generate_dedup_dataset;

pub use utils::generate_incompressible_data;

pub use utils::generate_mixed_size_blocks;

pub use utils::generate_pattern_blocks;

pub use utils::generate_random_block;

pub use utils::generate_random_blocks;

pub use utils::group_blocks_by_size;

pub use utils::sample_blocks;

pub use utils::sort_blocks_by_size_asc;

pub use utils::sort_blocks_by_size_desc;

pub use utils::validate_block_integrity;

pub use utils::validate_blocks_batch;

pub use utils::BlockStatistics;

pub use vcs::Author;

pub use vcs::Commit;

pub use vcs::CommitBuilder;

pub use vcs::MergeResult;

pub use vcs::MergeStrategy;

pub use vcs::Ref;

pub use vcs::RefType;

pub use vcs::VersionControl;

pub use wal::fnv1a_32;

pub use wal::StorageWalEntry;

pub use wal::StorageWalStats;

pub use wal::StorageWriteAheadLog;

pub use wal::WalConfig;

pub use wal::WalEntry;

pub use wal::WalEntryKind;

pub use wal::WalError;

pub use wal::WalOp;

pub use wal::WalStats;

pub use wal::WalStatsSnapshot;

pub use wal::WriteAheadLog;

pub use workload::OperationMix;

pub use workload::SizeDistribution as WorkloadSizeDistribution;

pub use workload::WorkloadConfig;

pub use workload::WorkloadPattern;

pub use workload::WorkloadPresets;

pub use workload::WorkloadResult;

pub use workload::WorkloadSimulator;

pub use write_buffer::BufferConfig;

pub use write_buffer::BufferedEntry;

pub use write_buffer::FlushResult;

pub use write_buffer::StorageWriteAheadBuffer;

pub use write_buffer::WriteOp;

pub use write_journal::fnv1a as journal_fnv1a;

pub use write_journal::JournalCursor;

pub use write_journal::JournalEntry;

pub use write_journal::JournalEntryKind;

pub use write_journal::JournalStats;

pub use write_journal::StorageWriteJournal;

pub use hotspot_detector::AccessEvent as HotspotAccessEvent;

pub use hotspot_detector::AccessType;

pub use hotspot_detector::BlockAccessRecord;

pub use hotspot_detector::DetectorConfig;

pub use hotspot_detector::HotspotDetectorStats;

pub use hotspot_detector::HotspotScore;

pub use hotspot_detector::StorageHotspotDetector;

pub use object_store::ObjectStoreStats;

pub use object_store::ObjectVersion;

pub use object_store::StorageObjectStore;

pub use object_store::StoredObject;

pub use blockstore_sharding::BlockRecord as BssBlockRecord;

pub use blockstore_sharding::BlockStoreGlobalStats;

pub use blockstore_sharding::BlockStoreSharding;

pub use blockstore_sharding::ShardKey;

pub use blockstore_sharding::ShardMetrics;

pub use blockstore_sharding::ShardingConfig;

pub use blockstore_sharding::ShardingError;

pub use migration_planner::MigrationDirection;

pub use migration_planner::MigrationRecord as PlannerMigrationRecord;

pub use migration_planner::MigrationStatus as PlannerMigrationStatus;

pub use migration_planner::MigrationTask;

pub use migration_planner::PlannerStats;

pub use migration_planner::StorageMigrationPlanner;

pub use migration_planner::StorageTier as MigrationStorageTier;

pub use transaction_log::StorageTransactionLog;

pub use transaction_log::Transaction;

pub use transaction_log::TransactionId;

pub use transaction_log::TransactionStatus;

pub use transaction_log::TxError;

pub use transaction_log::TxOperation;

pub use transaction_log::TxStats;

pub use block_compactor::BlockFragment;

pub use block_compactor::CompactionPlan;

pub use block_compactor::CompactionSegment;

pub use block_compactor::CompactorConfig;

pub use block_compactor::CompactorStats;

pub use block_compactor::StorageBlockCompactor;

pub use access_log::AccessLogEntry;

pub use access_log::AccessLogStats;

pub use access_log::AccessPattern as LogAccessPattern;

pub use access_log::LogConfig;

pub use access_log::LogOperation;

pub use access_log::StorageAccessLog;

pub use access_tracker::AccessRecord;

pub use access_tracker::AccessTracker as AtAccessTracker;

pub use access_tracker::TrackerConfig;

pub use access_tracker::TrackerStats;

pub use metadata_index::MetadataField;

pub use metadata_index::MetadataIndexEntry;

pub use metadata_index::MetadataIndexStats;

pub use metadata_index::MetadataQuery;

pub use metadata_index::MetadataQueryResult;

pub use metadata_index::MetadataSortField;

pub use metadata_index::StorageMetadataIndex;

pub use replication_manager::BlockReplicas;

pub use replication_manager::ReplicaInfo;

pub use replication_manager::ReplicaNode;

pub use replication_manager::ReplicationConfig as ReplicationManagerConfig;

pub use replication_manager::ReplicationError;

pub use replication_manager::ReplicationManagerStats;

pub use replication_manager::ReplicationState as ReplicaReplicationState;

pub use replication_manager::RmReplicaLocation;

pub use replication_manager::RmReplicationPolicy;

pub use replication_manager::RmReplicationStats;

pub use replication_manager::RmReplicationStatus;

pub use replication_manager::StorageReplicationManager;

pub use io_scheduler::IODirection;

pub use io_scheduler::IOPriority;

pub use io_scheduler::IORequest;

pub use io_scheduler::IOSchedulerStats;

pub use io_scheduler::SchedulerConfig;

pub use io_scheduler::StorageIOScheduler;

pub use encryption_layer::CipherMode;

pub use encryption_layer::EncryptedBlock;

pub use encryption_layer::EncryptionLayerConfig;

pub use encryption_layer::EncryptionLayerStats;

pub use encryption_layer::StorageEncryptionLayer;

pub use storage_encryption_layer::SelCipher;

pub use storage_encryption_layer::SelEncryptedBlockRecord;

pub use storage_encryption_layer::SelEncryptionConfig;

pub use storage_encryption_layer::SelEncryptionStats;

pub use storage_encryption_layer::StorageEncryptionLayer as SelStorageEncryptionLayer;

pub use event_log::EventLogStats;

pub use event_log::EventSeverity;

pub use event_log::EventType;

pub use event_log::StorageEvent;

pub use event_log::StorageEventLog;

pub use storage_benchmark::BenchmarkConfig;

pub use storage_benchmark::BenchmarkOp;

pub use storage_benchmark::BenchmarkResult;

pub use storage_benchmark::BenchmarkStats;

pub use storage_benchmark::LatencySample;

pub use storage_benchmark::StorageBenchmark;

pub use block_cache_manager::BcmCacheConfig;

pub use block_cache_manager::BcmCacheStats;

pub use block_cache_manager::BcmCachedBlock;

pub use block_cache_manager::BlockCacheManager;

pub use block_cache_manager::CacheTier;

pub use block_cache_manager::EvictionPolicy;

pub use storage_quota_manager::ObjectRecord;

pub use storage_quota_manager::QuotaError;

pub use storage_quota_manager::QuotaNamespace;

pub use storage_quota_manager::QuotaPolicy;

pub use storage_quota_manager::QuotaStats;

pub use storage_quota_manager::SqmEvictionStrategy;

pub use storage_quota_manager::SqmQuotaEntry;

pub use storage_quota_manager::SqmQuotaViolation;

pub use storage_quota_manager::StorageQuotaManager;

pub use content_addressed_archive::compute_cid as caa_compute_cid;

pub use content_addressed_archive::ArchiveBlock;

pub use content_addressed_archive::ArchiveConfig;

pub use content_addressed_archive::ArchiveEntry;

pub use content_addressed_archive::ArchiveError;

pub use content_addressed_archive::ArchiveIndex;

pub use content_addressed_archive::ArchiveStats;

pub use content_addressed_archive::ContentAddressedArchive;

pub use storage_event_log::event_checksum;

pub use storage_event_log::event_cid;

pub use storage_event_log::sel_fnv1a_64;

pub use storage_event_log::EventAggregation;

pub use storage_event_log::EventFilter;

pub use storage_event_log::EventId;

pub use storage_event_log::EventLogConfig;

pub use storage_event_log::EventLogError;

pub use storage_event_log::EventQuery;

pub use storage_event_log::RetentionPolicy;

pub use storage_event_log::SelEventLogStats;

pub use storage_event_log::SelEventType;

pub use storage_event_log::SelStorageEvent;

pub use storage_event_log::SelStorageEventLog;

pub use storage_event_log::StorageEventKind;

pub use object_lifecycle::LifecycleError;

pub use object_lifecycle::ManagedObject;

pub use object_lifecycle::ObjectLifecycleManager;

pub use object_lifecycle::OlmLifecycleAction;

pub use object_lifecycle::OlmLifecycleState;

pub use object_lifecycle::OlmLifecycleStats;

pub use object_lifecycle::OlmRetentionRule;

pub use prefetch_engine::CoAccessPair;

pub use prefetch_engine::PeAccessEvent;

pub use prefetch_engine::PeAccessPattern;

pub use prefetch_engine::PeAccessType;

pub use prefetch_engine::PeConfig;

pub use prefetch_engine::PePrefetchHint;

pub use prefetch_engine::PePrefetchStats;

pub use prefetch_engine::StoragePrefetchEngine;

pub use block_fragment_store::bfs_fnv1a_32;

pub use block_fragment_store::AssembledBlock;

pub use block_fragment_store::BfsFragment;

pub use block_fragment_store::BlockFragmentStore;

pub use block_fragment_store::FragmentError;

pub use block_fragment_store::FragmentId;

pub use block_fragment_store::FragmentSet;

pub use block_fragment_store::FragmentSetState;

pub use block_fragment_store::FragmentStats;

pub use mirror_sync::fnv1a_64 as mirror_sync_fnv1a_64;

pub use mirror_sync::ConflictType;

pub use mirror_sync::MirrorId;

pub use mirror_sync::MirrorSyncStats;

pub use mirror_sync::MsConflictResolution;

pub use mirror_sync::MsSyncResult;

pub use mirror_sync::StorageMirrorSync;

pub use mirror_sync::SyncConflict;

pub use mirror_sync::SyncItem;

pub use mirror_sync::SyncOperation;

pub use mirror_sync::SyncPlan;

pub use checksum_engine::adler32 as ce_adler32;

pub use checksum_engine::blake3_256_simple as ce_blake3_256_simple;

pub use checksum_engine::crc32_iso as ce_crc32_iso;

pub use checksum_engine::djb2 as ce_djb2;

pub use checksum_engine::fnv1a_64 as ce_fnv1a_64;

pub use checksum_engine::murmur3_32 as ce_murmur3_32;

pub use checksum_engine::xxhash64_simple as ce_xxhash64_simple;

pub use checksum_engine::CeVerificationResult;

pub use checksum_engine::Checksum;

pub use checksum_engine::ChecksumAlgorithm;

pub use checksum_engine::ChecksumRecord as CeChecksumRecord;

pub use checksum_engine::ChecksumStats;

pub use checksum_engine::StorageChecksumEngine;

pub use wal_replay::ReplayPolicy as WrReplayPolicy;

pub use wal_replay::ReplayState as WrReplayState;

pub use wal_replay::ReplayStats as WrReplayStats;

pub use wal_replay::StorageWALReplay;

pub use wal_replay::WalEntry as WrWalEntry;

pub use wal_replay::WalEntryType as WrWalEntryType;

pub use wal_replay::WalStats as WrWalStats;

pub use block_access_optimizer::AccessEvent as BaoAccessEvent;

pub use block_access_optimizer::AccessPattern as BaoAccessPattern;

pub use block_access_optimizer::BlockAccessOptimizer;

pub use block_access_optimizer::CoAccessPair as BaoCoAccessPair;

pub use block_access_optimizer::OptimizerConfig as BaoOptimizerConfig;

pub use block_access_optimizer::OptimizerStats;

pub use block_access_optimizer::PrefetchRecommendation;

pub use block_index_rebuild::BlockIndexRebuild;

pub use block_index_rebuild::BlockScanEntry;

pub use block_index_rebuild::IndexEntry as BirIndexEntry;

pub use block_index_rebuild::RebuildConfig;

pub use block_index_rebuild::RebuildPhase;

pub use block_index_rebuild::RebuildProgress;

pub use block_index_rebuild::RebuildStats;

pub use object_version_store::ObjectVersionStore;

pub use object_version_store::OvsGcPolicy;

pub use object_version_store::OvsObjectVersion;

pub use object_version_store::VersionBranch;

pub use object_version_store::VersionQuery;

pub use object_version_store::VersionStoreConfig;

pub use object_version_store::VsError;

pub use object_version_store::VsStats;

pub use merkle_proof_verifier::MerkleHash;

pub use merkle_proof_verifier::MerkleLeaf;

pub use merkle_proof_verifier::MerkleProof;

pub use merkle_proof_verifier::MerkleProofVerifier;

pub use merkle_proof_verifier::ProofStep;

pub use merkle_proof_verifier::TreeStats as MerkleTreeStats;

pub use merkle_proof_verifier::UpdateProof as MerkleUpdateProof;

pub use merkle_proof_verifier::VerifierError as MerkleVerifierError;

pub use storage_shard_balancer::fnv1a_64 as ssb_fnv1a_64;

pub use storage_shard_balancer::xorshift64 as ssb_xorshift64;

pub use storage_shard_balancer::BalancerConfig;

pub use storage_shard_balancer::BalancerError;

pub use storage_shard_balancer::RebalanceOp;

pub use storage_shard_balancer::RebalancePolicy;

pub use storage_shard_balancer::ShardAssignment;

pub use storage_shard_balancer::ShardNode;

pub use storage_shard_balancer::SsbBalancerStats;

pub use storage_shard_balancer::StorageShardBalancer;

pub use storage_quota_enforcer::EnforcementPolicy;

pub use storage_quota_enforcer::EnforcerConfig;

pub use storage_quota_enforcer::GrowthForecast;

pub use storage_quota_enforcer::NamespaceId;

pub use storage_quota_enforcer::QuotaLimit;

pub use storage_quota_enforcer::QuotaUsage;

pub use storage_quota_enforcer::QuotaViolation as SqeQuotaViolation;

pub use storage_quota_enforcer::SqeStorageQuotaEnforcer;

pub use storage_quota_enforcer::UsageSample;

pub use storage_quota_enforcer::ViolationKind;

pub use block_deduplicator::fnv1a_64 as bdd_fnv1a_64;

pub use block_deduplicator::BlockDeduplicator;

pub use block_deduplicator::Chunk as BddChunk;

pub use block_deduplicator::ChunkHash;

pub use block_deduplicator::ChunkRef;

pub use block_deduplicator::ChunkingConfig as BddChunkingConfig;

pub use block_deduplicator::DeduplicationStats as BddDeduplicationStats;

pub use block_deduplicator::DeduplicatorError;

pub use block_deduplicator::ObjectManifest;

pub use object_storage_tiering::ObjectStorageTiering;

pub use object_storage_tiering::OstStorageTier;

pub use object_storage_tiering::OstTierConfig;

pub use object_storage_tiering::OstTierPolicy;

pub use object_storage_tiering::OstTierTransition;

pub use object_storage_tiering::TieredObject;

pub use object_storage_tiering::TieringError;

pub use object_storage_tiering::TieringStats;

pub use write_ahead_log::fnv1a_64 as wal2_fnv1a_64;

pub use write_ahead_log::RecoveryResult;

pub use write_ahead_log::Transaction as WalTransaction;

pub use write_ahead_log::TxState;

pub use write_ahead_log::WalConfig as WalWalConfig;

pub use write_ahead_log::WalEntry as WalWalEntry;

pub use write_ahead_log::WalError as WalWalError;

pub use write_ahead_log::WalOpType;

pub use write_ahead_log::WalStats as WalWalStats;

pub use write_ahead_log::WalWriteAheadLog;

pub use write_ahead_log::WAL_MAGIC;

pub use storage_compression_pipeline::delta_decode;

pub use storage_compression_pipeline::delta_encode;

pub use storage_compression_pipeline::fnv1a_64 as scp_fnv1a_64;

pub use storage_compression_pipeline::lz77_decode;

pub use storage_compression_pipeline::lz77_encode;

pub use storage_compression_pipeline::rle_decode;

pub use storage_compression_pipeline::rle_encode;

pub use storage_compression_pipeline::xor_transform;

pub use storage_compression_pipeline::CompressedBlock;

pub use storage_compression_pipeline::CompressionStage;

pub use storage_compression_pipeline::ScpCompressionAlgorithm;

pub use storage_compression_pipeline::ScpPipelineConfig;

pub use storage_compression_pipeline::ScpPipelineError;

pub use storage_compression_pipeline::ScpPipelineStats;

pub use storage_compression_pipeline::ScpStorageCompressionPipeline;

pub use storage_access_controller::AccessDecision;

pub use storage_access_controller::AclConfig;

pub use storage_access_controller::AclError;

pub use storage_access_controller::AclStats;

pub use storage_access_controller::Permission;

pub use storage_access_controller::PolicyEffect;

pub use storage_access_controller::ResourcePolicy;

pub use storage_access_controller::SacAuditEntry;

pub use storage_access_controller::SacRole;

pub use storage_access_controller::StorageAccessController;

pub use storage_access_controller::SubjectAttributes;

pub use object_integrity_checker::CheckerConfig;

pub use object_integrity_checker::CheckerError;

pub use object_integrity_checker::CheckerStats;

pub use object_integrity_checker::IntegrityHash;

pub use object_integrity_checker::IntegrityLevel;

pub use object_integrity_checker::ObjectIntegrityChecker;

pub use object_integrity_checker::OicIntegrityStatus;

pub use object_integrity_checker::OicObjectRecord;

pub use object_integrity_checker::OicVerificationResult;

pub use storage_replication_manager::ReplicaTarget as SrmReplicaTargetDirect;

pub use storage_replication_manager::ReplicationPolicy as SrmReplicationPolicyDirect;

pub use storage_replication_manager::SrmError as SrmReplicationError;

pub use storage_replication_manager::SrmReplicaTarget;

pub use storage_replication_manager::SrmReplicationConfig;

pub use storage_replication_manager::SrmReplicationOp;

pub use storage_replication_manager::SrmReplicationPolicy;

pub use storage_replication_manager::SrmReplicationStats;

pub use storage_replication_manager::StorageReplicationManager as SrmStorageReplicationManager;

pub use storage_query_planner::SqpCostModel;

pub use storage_query_planner::SqpPlanStep;

pub use storage_query_planner::SqpPlannerConfig;

pub use storage_query_planner::SqpPlannerStats;

pub use storage_query_planner::SqpQuery;

pub use storage_query_planner::SqpQueryPlan;

pub use storage_query_planner::StorageQueryPlanner;

pub use storage_snapshot_manager::fnv1a_64 as ssm_fnv1a_64;

pub use storage_snapshot_manager::CoWMapping;

pub use storage_snapshot_manager::Page;

pub use storage_snapshot_manager::PageId;

pub use storage_snapshot_manager::SnapshotConfig as CowSnapshotConfig;

pub use storage_snapshot_manager::SnapshotDiff as CowSnapshotDiff;

pub use storage_snapshot_manager::SnapshotError as CowSnapshotError;

pub use storage_snapshot_manager::SnapshotId as CowSnapshotId;

pub use storage_snapshot_manager::SnapshotMetadata;

pub use storage_snapshot_manager::SnapshotStats as CowSnapshotStats;

pub use storage_snapshot_manager::StorageSnapshotManager as CowStorageSnapshotManager;

Modules

access_log

Append-only storage access log with configurable retention, pattern analysis, and export.

access_logger

Storage access logger for audit trails and access pattern analysis.

access_predictor

Storage access predictor for proactive prefetching and cache warming.

access_tracker

Access frequency and recency tracking for storage tiering and eviction decisions.

analyzer

Comprehensive storage analysis and optimization tools

arm_profiler

ARM Performance Profiler

audit_trail

Immutable audit trail for storage operations.

auto_tuner

Automatic configuration tuning based on workload patterns

batch

Batch operation utilities for efficient bulk processing

block_access_optimizer

Block Access Optimizer — learns co-access patterns, predicts future accesses, and generates prefetch recommendations to minimize storage latency.

block_cache

In-memory LRU/LFU block cache with TTL-based and size-based eviction.

block_cache_manager

Multi-tier, policy-driven block cache manager for IPFS-style content-addressed storage.

block_compactor

Storage Block Compactor

block_deduplicator

Content-aware block deduplication with variable-length chunking.

block_fragment_store

Block Fragment Store

block_garbage_collector

Block-level garbage collector for IPFRS content-addressed storage.

block_index

Secondary index over stored blocks for accelerated queries.

block_index_rebuild

Block Index Rebuild — index reconstruction engine for IPFRS block storage.

block_manifest

Storage Block Manifest — catalogs all blocks in a partition with metadata for fast lookup, consistency checking, and export/import.

block_migration_planner

Block migration planner: schedules and tracks data movement between storage nodes/tiers.

block_packer

Storage Block Packer — packs multiple small blocks into larger “pack files” to reduce storage overhead and improve sequential read performance, similar to Git’s packfile format.

block_stream

Block stream iterator with backpressure for large CAR/tensor exports.

block_validator

Storage block validator with multiple hash and integrity checks.

block_verifier

Storage block integrity verifier using FNV-1a checksums.

blockstore

Block storage implementation

blockstore_sharding

Sharded block storage system for distributed content-addressed blocks.

bloom

Bloom filter for probabilistic block existence checks.

cache

In-memory block cache — L1/L2 hot cache above a backing BlockStore

car

CAR (Content Addressable aRchive) format support.

checksum_engine

Multi-algorithm checksum computation and verification for storage integrity.

chunk_manager

Storage chunk manager for splitting large objects into fixed-size chunks.

circuit_breaker

Circuit Breaker pattern for fault tolerance

cluster

Cluster coordinator for multi-node RAFT deployments.

coalesce

Write coalescing for batching similar writes

cold_storage

Cold storage tier management for archiving infrequently accessed blocks.

compaction

Sled compaction scheduling with lock-free atomics.

compaction_advisor

Compaction advisor for data-driven storage compaction recommendations.

compactor

Storage Compactor

compression_advisor

Block compression advisor — analyses access patterns and sizes to recommend optimal compression codecs and tracks achieved compression ratios.

compression_pipeline

StorageCompressionPipeline — configurable multi-stage compression pipeline.

compression_registry

StorageCompressionRegistry — tracks codec registrations, usage statistics, and provides codec selection recommendations based on data characteristics.

content_addressable_cache

Content-Addressable Cache with LRU eviction and TTL support.

content_addressed_archive

Content-Addressed Archive — an append-only, in-memory store for immutable data blobs.

content_addressed_cache_v2

Content-Addressed Cache V2 — multi-tier eviction with admission control.

content_dedup_index

Content deduplication index for tracking and merging identical content blocks.

corruption_repair

Block corruption detection and repair strategies.

cost_analytics

Cost Analytics for storage optimization

cost_estimator

Storage Cost Estimator

data_integrity_auditor

Data Integrity Auditor — proactive continuous verification of stored blocks.

data_integrity_checker

Data Integrity Checker - Deep integrity verification for stored blocks

datacenter

Multi-Datacenter Support for Distributed Storage

dedup

Block deduplication using content-defined chunking.

dedup_tracker

Block deduplication tracker for IPFRS storage.

deduplication_pipeline

Multi-stage block deduplication pipeline combining exact-match and content-based deduplication.

diagnostics

Storage diagnostics and health monitoring utilities

encryption_layer

Storage Encryption Layer

event_log

Structured event logging for storage operations.

eventual_consistency

Eventual consistency support for distributed storage.

eviction_policy

Pluggable cache eviction policy engine for IPFRS storage tier management.

eviction_simulator

Cache eviction policy simulator.

exporters

Metric exporters for integration with monitoring systems

fragmentation_analyzer

Storage Fragmentation Analyzer

garbage_collector

Mark-and-sweep garbage collector for content-addressed storage.

gc

Garbage Collection for block storage.

gc_planner

GC Planner — analyzes block liveness and produces ordered deletion plans.

gradient

Gradient and tensor storage with delta encoding

health

Health check system for storage backends

health_monitor

Storage Health Monitor

heatmap_tracker

Storage Heatmap Tracker

helpers

Helper functions for creating storage stacks

hotspot_detector

Storage hotspot detector using sliding window access counting and exponential decay scoring.

index_builder

Secondary index over block metadata enabling fast filtered queries.

index_recovery

Storage index recovery from corrupt or missing state by scanning raw block data.

integrity_checker

Block Integrity Checker

integrity_scanner

Block Integrity Scanner

io_scheduler

I/O request scheduling with deadline and priority ordering.

lifecycle

Lifecycle Policies for automatic data management

memory

In-memory block store for testing

merkle_proof_verifier

Merkle tree construction and proof verification.

metadata_index

Secondary index over block metadata fields for efficient filtered queries.

metrics

Storage metrics and observability

metrics_collector

Storage metrics collector with sliding window aggregation and alerting.

migration

Storage migration utilities

migration_planner

Storage migration planner: schedules and tracks block movements between tiers.

mirror_sync

Bidirectional sync between storage mirrors with conflict detection, resolution, and audit.

object_integrity_checker

Object Integrity Checker — multi-level hash-based object integrity verification.

object_lifecycle

Object Lifecycle Manager — Policy-driven object lifecycle management

object_storage_tiering

ObjectStorageTiering — hierarchical object storage tiering (hot/warm/cold).

object_store

High-level object store abstraction over raw block storage.

object_version_store

Versioned object store with full history, branching, and garbage collection.

otel

OpenTelemetry integration for distributed tracing

pinning

Pin management for preventing garbage collection of important blocks.

pool

Storage Pool Manager for multi-backend routing

prefetch

Predictive Prefetching for intelligent block preloading

prefetch_engine

StoragePrefetchEngine — Intelligent prefetch engine that learns access patterns and pre-warms the cache by predicting future access.

profiler

Comprehensive storage profiling and optimization

profiling

Advanced performance profiling utilities

prometheus

Prometheus metrics exporter

query_optimizer

Query optimizer for storage operations

quota

Quota Management for per-tenant storage limits

quota_enforcer

Storage Quota Enforcer — per-namespace soft/hard limit enforcement

quota_manager

Storage Quota Manager — per-namespace quota tracking and enforcement

quota_registry

Storage Quota Registry

raft

RAFT Consensus Protocol Implementation

rate_limit

Rate limiting for controlling request rates to backends

read_ahead

Read-Ahead Scheduler for sequential block prefetching

replication

Block replication and synchronization

replication_manager

Storage Replication Manager

replication_tracker

Storage Replication Tracker

retention_engine

Retention Policy Engine for IPFRS block storage.

retention_policy

Storage Retention Policy Engine for IPFRS block storage.

retry

Retry logic with exponential backoff and jitter

safetensors

Safetensors format support for efficient model storage

snapshot_diff

Storage snapshot diff — computes and represents differences between two point-in-time storage snapshots.

snapshot_differ

Snapshot differ — computes the minimal set of block operations needed to transform one storage snapshot into another.

snapshot_manager

StorageSnapshotManager — Point-in-time snapshot management for storage state with incremental deltas and restoration.

storage_access_controller

Role-based and attribute-based access control (RBAC + ABAC) for storage operations.

storage_benchmark

Storage performance benchmarking utilities

storage_compression_pipeline

StorageCompressionPipeline — configurable multi-stage compression pipeline.

storage_encryption_layer

Storage Encryption Layer (storage_encryption_layer)

storage_event_log

Production-grade structured event log for storage operations.

storage_health_monitor

Storage Health Monitor

storage_metrics_collector

Comprehensive metrics collection and aggregation system for storage operations.

storage_query_planner

Storage Query Planner — optimizes storage access patterns for batch read/write workloads.

storage_quota_enforcer

Storage Quota Enforcer — per-namespace quota enforcement with forecasting

storage_quota_manager

Per-namespace storage quota enforcement with usage tracking and eviction triggers.

storage_replication_manager

StorageReplicationManager — production-grade replication orchestration for IPFS blocks.

storage_shard_balancer

StorageShardBalancer — consistent-hashing-based shard management and rebalancing.

storage_snapshot_manager

Copy-on-Write storage snapshot manager.

streaming

Streaming interfaces for large block operations.

tier_balancer

StorageTierBalancer — monitors utilization across storage tiers and generates rebalancing plans that move blocks to meet target utilization ratios.

tier_manager

StorageTierManager — policy-driven hot/warm/cold/archive tier classification.

tier_migration_engine

StorageTierMigrator — intelligent block migration engine for storage tiers.

tier_migrator

Block tier migration manager.

tiering

Hot/cold storage tiering with access tracking.

traits

Storage traits

transaction_log

ACID-like transaction logging with commit, rollback, and replay support.

transport

Network transport abstraction for distributed RAFT.

ttl

Time-To-Live (TTL) support for automatic block expiration

utils

Utility functions for testing, benchmarking, and batch operations

vcs

Version Control System for Differentiable Storage

wal

Write-Ahead Log (WAL) for crash-recovery of block store writes.

wal_replay

WAL Replay Engine — crash recovery and state reconstruction from WAL entries.

workload

Workload simulation and generation for testing and benchmarking

write_ahead_log

Production-grade Write-Ahead Log (WAL) for crash recovery and durability.

write_buffer

Storage Write-Ahead Buffer — buffers incoming writes in memory before flushing to the underlying storage backend.

write_journal

Storage Write Journal — append-only record of all storage mutations for crash recovery, audit, and replication purposes.