Crate ipfrs_storage 

IPFRS Storage - Block storage and retrieval

This crate provides the storage layer for IPFRS including:

• Sled, ParityDB, and in-memory block stores
• LRU and tiered caching
• Bloom filter for fast existence checks
• Streaming interface for large blocks
• Content-defined chunking and deduplication
• Transparent block compression (Zstd, Lz4, Snappy)
• Pin management for preventing GC
• Hot/cold tiering with access tracking
• Garbage collection (mark-and-sweep)
• CAR format export/import for backups
• Encryption at rest (ChaCha20-Poly1305, AES-256-GCM)
• Version Control System for differentiable storage (Git for Tensors)
• RAFT consensus protocol for distributed storage
• Network transport abstraction (TCP, QUIC with TLS) for multi-node RAFT clusters
• Cluster coordinator with automatic failover and re-election
• Multi-datacenter support with latency-aware routing and cross-datacenter replication
• Eventual consistency with version vectors and conflict resolution
• GraphQL query interface for flexible metadata querying
• ARM profiler with NEON SIMD optimization and low-power tuning
• Production-grade metrics and observability for monitoring (Prometheus, OpenTelemetry)
• Circuit breaker pattern for fault-tolerant external service calls
• Unified health check system for liveness and readiness monitoring
• TTL support for automatic block expiration
• Advanced retry logic with exponential backoff and jitter
• Optimized S3 multipart uploads for large blocks
• Rate limiting for controlling request rates to backends
• Write coalescing for batching similar writes
• Workload simulation for testing and benchmarking
• Automatic configuration tuning based on workload patterns
• Comprehensive profiling system with comparative analysis and regression detection
• Storage pool manager for multi-backend routing with intelligent load balancing
• Quota management system for per-tenant storage limits and bandwidth control
• Lifecycle policies for automatic data tiering, archival, and expiration
• Predictive prefetching with access pattern learning and adaptive depth control
• Cost analytics and optimization for cloud storage (AWS, Azure, GCP)

Re-exports

pub use analyzer::Category;

pub use analyzer::Difficulty;

pub use analyzer::OperationStats;

pub use analyzer::OptimizationRecommendation;

pub use analyzer::Priority;

pub use analyzer::SizeDistribution;

pub use analyzer::StorageAnalysis;

pub use analyzer::StorageAnalyzer;

pub use analyzer::WorkloadCharacterization;

pub use analyzer::WorkloadType;

pub use arm_profiler::hash_block;

pub use arm_profiler::ArmFeatures;

pub use arm_profiler::ArmPerfCounter;

pub use arm_profiler::ArmPerfReport;

pub use arm_profiler::LowPowerBatcher;

pub use arm_profiler::PowerProfile;

pub use arm_profiler::PowerStats;

pub use auto_tuner::AutoTuner;

pub use auto_tuner::AutoTunerConfig;

pub use auto_tuner::TuningPresets;

pub use auto_tuner::TuningRecommendation;

pub use auto_tuner::TuningReport;

pub use batch::batch_delete;

pub use batch::batch_get;

pub use batch::batch_has;

pub use batch::batch_put;

pub use batch::BatchConfig;

pub use batch::BatchResult;

pub use blockstore::BlockStoreConfig;

pub use blockstore::SledBlockStore;

pub use bloom::BloomBlockStore;

pub use bloom::BloomConfig;

pub use bloom::BloomFilter;

pub use bloom::BloomStats;

pub use cache::BlockCache;

pub use cache::CacheStats;

pub use cache::CachedBlockStore;

pub use cache::TieredBlockCache;

pub use cache::TieredCacheStats;

pub use cache::TieredCachedBlockStore;

pub use car::export_to_car;

pub use car::import_from_car;

pub use car::CarHeader;

pub use car::CarReadStats;

pub use car::CarReader;

pub use car::CarWriteStats;

pub use car::CarWriter;

pub use circuit_breaker::CircuitBreaker;

pub use circuit_breaker::CircuitState;

pub use circuit_breaker::CircuitStats;

pub use cluster::ClusterConfig;

pub use cluster::ClusterCoordinator;

pub use cluster::ClusterStats;

pub use cluster::NodeHealth;

pub use cluster::NodeInfo;

pub use coalesce::CoalesceConfig;

pub use coalesce::CoalesceStats;

pub use coalesce::CoalescingBlockStore;

pub use cost_analytics::CloudProvider;

pub use cost_analytics::CostAnalyzer;

pub use cost_analytics::CostBreakdown;

pub use cost_analytics::CostProjection;

pub use cost_analytics::CostTier;

pub use cost_analytics::TierCostModel;

pub use cost_analytics::TierOption;

pub use cost_analytics::TierRecommendation;

pub use datacenter::CrossDcStats;

pub use datacenter::Datacenter;

pub use datacenter::DatacenterId;

pub use datacenter::LatencyAwareSelector;

pub use datacenter::MultiDatacenterCoordinator;

pub use datacenter::Region;

pub use datacenter::ReplicationPolicy;

pub use dedup::ChunkingConfig;

pub use dedup::DedupBlockStore;

pub use dedup::DedupStats;

pub use diagnostics::BenchmarkComparison;

pub use diagnostics::DiagnosticsReport;

pub use diagnostics::HealthMetrics;

pub use diagnostics::PerformanceMetrics;

pub use diagnostics::StorageDiagnostics;

pub use eventual_consistency::ConflictResolution;

pub use eventual_consistency::ConsistencyLevel;

pub use eventual_consistency::EventualStore;

pub use eventual_consistency::EventualStoreStats;

pub use eventual_consistency::VersionVector;

pub use eventual_consistency::VersionedValue;

pub use exporters::BatchExporter;

pub use exporters::ExportFormat;

pub use exporters::MetricExporter;

pub use gc::GarbageCollector;

pub use gc::GcConfig;

pub use gc::GcPolicy;

pub use gc::GcResult;

pub use gc::GcScheduler;

pub use gc::GcStats;

pub use gc::GcStatsSnapshot;

pub use gradient::CompressionStats;

pub use gradient::DeltaEncoder;

pub use gradient::GradientData;

pub use gradient::GradientStore;

pub use gradient::ProvenanceMetadata;

pub use health::AggregateHealthResult;

pub use health::DetailedHealthStatus;

pub use health::HealthCheck;

pub use health::HealthCheckResult;

pub use health::HealthChecker;

pub use health::HealthStatus;

pub use health::SimpleHealthCheck;

pub use helpers::blockchain_stack;

pub use helpers::cache_stack;

pub use helpers::cdn_edge_stack;

pub use helpers::coalescing_memory_stack;

pub use helpers::deduplicated_production_stack;

pub use helpers::development_stack;

pub use helpers::embedded_stack;

pub use helpers::ingestion_stack;

pub use helpers::iot_stack;

pub use helpers::media_streaming_stack;

pub use helpers::memory_stack;

pub use helpers::ml_model_stack;

pub use helpers::monitored_production_stack;

pub use helpers::production_stack;

pub use helpers::read_optimized_stack;

pub use helpers::resilient_stack;

pub use helpers::testing_stack;

pub use helpers::ttl_production_stack;

pub use helpers::write_optimized_stack;

pub use helpers::StorageStackBuilder;

pub use lifecycle::BlockMetadata as LifecycleBlockMetadata;

pub use lifecycle::LifecycleAction;

pub use lifecycle::LifecycleActionResult;

pub use lifecycle::LifecycleCondition;

pub use lifecycle::LifecyclePolicyConfig;

pub use lifecycle::LifecyclePolicyManager;

pub use lifecycle::LifecycleRule;

pub use lifecycle::LifecycleStatsSnapshot;

pub use lifecycle::StorageTier;

pub use memory::MemoryBlockStore;

pub use metrics::MetricsBlockStore;

pub use metrics::StorageMetrics;

pub use migration::estimate_migration;

pub use migration::migrate_storage;

pub use migration::migrate_storage_batched;

pub use migration::migrate_storage_verified;

pub use migration::migrate_storage_with_progress;

pub use migration::validate_migration;

pub use migration::MigrationConfig;

pub use migration::MigrationEstimate;

pub use migration::MigrationStats;

pub use migration::StorageMigrator;

pub use otel::OtelBlockStore;

pub use paritydb::ParityDbBlockStore;

pub use paritydb::ParityDbConfig;

pub use paritydb::ParityDbPreset;

pub use pinning::PinInfo;

pub use pinning::PinManager;

pub use pinning::PinSet;

pub use pinning::PinStatsSnapshot;

pub use pinning::PinType;

pub use pool::BackendConfig;

pub use pool::BackendId;

pub use pool::PoolConfig;

pub use pool::PoolStats;

pub use pool::RoutingStrategy;

pub use pool::StoragePool;

pub use prefetch::AccessPattern;

pub use prefetch::PredictivePrefetcher;

pub use prefetch::PrefetchConfig;

pub use prefetch::PrefetchPrediction;

pub use prefetch::PrefetchStatsSnapshot;

pub use profiler::ComparativeProfiler;

pub use profiler::ComparisonReport;

pub use profiler::ProfileConfig;

pub use profiler::ProfileReport;

pub use profiler::RegressionDetector;

pub use profiler::RegressionResult;

pub use profiler::StorageProfiler;

pub use profiling::BatchProfiler;

pub use profiling::LatencyHistogram;

pub use profiling::PerformanceProfiler;

pub use profiling::ThroughputTracker;

pub use prometheus::PrometheusExporter;

pub use prometheus::PrometheusExporterBuilder;

pub use query_optimizer::OptimizerConfig;

pub use query_optimizer::QueryLogEntry;

pub use query_optimizer::QueryOptimizer;

pub use query_optimizer::QueryPlan;

pub use query_optimizer::QueryStrategy;

pub use query_optimizer::Recommendation;

pub use query_optimizer::RecommendationCategory;

pub use query_optimizer::RecommendationPriority;

pub use quota::QuotaBlockStore;

pub use quota::QuotaConfig;

pub use quota::QuotaManager;

pub use quota::QuotaManagerConfig;

pub use quota::QuotaReport;

pub use quota::QuotaStatus;

pub use quota::QuotaUsageSnapshot;

pub use quota::TenantId;

pub use quota::ViolationType;

pub use raft::AppendEntriesRequest;

pub use raft::AppendEntriesResponse;

pub use raft::Command;

pub use raft::LogEntry;

pub use raft::LogIndex;

pub use raft::NodeId;

pub use raft::NodeState;

pub use raft::RaftConfig;

pub use raft::RaftNode;

pub use raft::RaftStats;

pub use raft::RequestVoteRequest;

pub use raft::RequestVoteResponse;

pub use raft::Term;

pub use rate_limit::RateLimitAlgorithm;

pub use rate_limit::RateLimitConfig;

pub use rate_limit::RateLimitStats;

pub use rate_limit::RateLimiter;

pub use replication::ConflictStrategy;

pub use replication::ReplicationManager;

pub use replication::ReplicationState;

pub use replication::Replicator;

pub use replication::SyncResult;

pub use replication::SyncStrategy;

pub use retry::BackoffStrategy;

pub use retry::JitterType;

pub use retry::RetryPolicy;

pub use retry::RetryStats;

pub use retry::Retryable;

pub use safetensors::ChunkConfig;

pub use safetensors::ChunkedTensor;

pub use safetensors::DType;

pub use safetensors::ModelStats;

pub use safetensors::SafetensorsHeader;

pub use safetensors::SafetensorsManifest;

pub use safetensors::SafetensorsStore;

pub use safetensors::TensorInfo;

pub use streaming::BlockReader;

pub use streaming::ByteRange;

pub use streaming::PartialBlock;

pub use streaming::StreamConfig;

pub use streaming::StreamingBlockStore;

pub use streaming::StreamingWriter;

pub use tiering::AccessStats;

pub use tiering::AccessTracker;

pub use tiering::Tier;

pub use tiering::TierConfig;

pub use tiering::TierStatsSnapshot;

pub use tiering::TieredStore;

pub use traits::BlockStore as BlockStoreTrait;

pub use transport::InMemoryTransport;

pub use transport::Message as TransportMessage;

pub use transport::TcpTransport;

pub use transport::Transport;

pub use transport::TransportConfig;

pub use ttl::TtlBlockStore;

pub use ttl::TtlCleanupResult;

pub use ttl::TtlConfig;

pub use ttl::TtlStats;

pub use utils::compute_cid;

pub use utils::compute_total_size;

pub use utils::create_block;

pub use utils::create_blocks_batch;

pub use utils::deduplicate_blocks;

pub use utils::estimate_compression_ratio;

pub use utils::extract_cids;

pub use utils::filter_blocks_by_size;

pub use utils::find_duplicates;

pub use utils::generate_compressible_blocks;

pub use utils::generate_compressible_data;

pub use utils::generate_dedup_dataset;

pub use utils::generate_incompressible_data;

pub use utils::generate_mixed_size_blocks;

pub use utils::generate_pattern_blocks;

pub use utils::generate_random_block;

pub use utils::generate_random_blocks;

pub use utils::group_blocks_by_size;

pub use utils::sample_blocks;

pub use utils::sort_blocks_by_size_asc;

pub use utils::sort_blocks_by_size_desc;

pub use utils::validate_block_integrity;

pub use utils::validate_blocks_batch;

pub use utils::BlockStatistics;

pub use vcs::Author;

pub use vcs::Commit;

pub use vcs::CommitBuilder;

pub use vcs::MergeResult;

pub use vcs::MergeStrategy;

pub use vcs::Ref;

pub use vcs::RefType;

pub use vcs::VersionControl;

pub use workload::OperationMix;

pub use workload::SizeDistribution as WorkloadSizeDistribution;

pub use workload::WorkloadConfig;

pub use workload::WorkloadPattern;

pub use workload::WorkloadPresets;

pub use workload::WorkloadResult;

pub use workload::WorkloadSimulator;

Modules

analyzer

Comprehensive storage analysis and optimization tools

arm_profiler

ARM Performance Profiler

auto_tuner

Automatic configuration tuning based on workload patterns

batch

Batch operation utilities for efficient bulk processing

blockstore

Block storage implementation using Sled

bloom

Bloom filter for probabilistic block existence checks.

cache

In-memory block cache

car

CAR (Content Addressable aRchive) format support.

circuit_breaker

Circuit Breaker pattern for fault tolerance

cluster

Cluster coordinator for multi-node RAFT deployments.

coalesce

Write coalescing for batching similar writes

cost_analytics

Cost Analytics for storage optimization

datacenter

Multi-Datacenter Support for Distributed Storage

dedup

Block deduplication using content-defined chunking.

diagnostics

Storage diagnostics and health monitoring utilities

eventual_consistency

Eventual consistency support for distributed storage.

exporters

Metric exporters for integration with monitoring systems

gc

Garbage Collection for block storage.

gradient

Gradient and tensor storage with delta encoding

health

Health check system for storage backends

helpers

Helper functions for creating storage stacks

lifecycle

Lifecycle Policies for automatic data management

memory

In-memory block store for testing

metrics

Storage metrics and observability

migration

Storage migration utilities

otel

OpenTelemetry integration for distributed tracing

paritydb

Block storage implementation using ParityDB

pinning

Pin management for preventing garbage collection of important blocks.

pool

Storage Pool Manager for multi-backend routing

prefetch

Predictive Prefetching for intelligent block preloading

profiler

Comprehensive storage profiling and optimization

profiling

Advanced performance profiling utilities

prometheus

Prometheus metrics exporter

query_optimizer

Query optimizer for storage operations

quota

Quota Management for per-tenant storage limits

raft

RAFT Consensus Protocol Implementation

rate_limit

Rate limiting for controlling request rates to backends

replication

Block replication and synchronization

retry

Retry logic with exponential backoff and jitter

safetensors

Safetensors format support for efficient model storage

streaming

Streaming interfaces for large block operations.

tiering

Hot/cold storage tiering with access tracking.

traits

Storage traits

transport

Network transport abstraction for distributed RAFT.

ttl

Time-To-Live (TTL) support for automatic block expiration

utils

Utility functions for testing, benchmarking, and batch operations

vcs

Version Control System for Differentiable Storage

workload

Workload simulation and generation for testing and benchmarking