Expand description
Gradient and tensor storage with delta encoding
Provides efficient storage for neural network gradients and tensors:
- Delta encoding (store changes only)
- Sparse gradient compression
- Provenance metadata tracking
- Integration with version control
§Example
ⓘ
use ipfrs_storage::{GradientStore, DeltaEncoder, SledBlockStore, BlockStoreConfig};
use std::sync::Arc;
use std::path::PathBuf;
// Create block store
let store = Arc::new(SledBlockStore::new(BlockStoreConfig {
path: PathBuf::from(".ipfrs/gradients"),
cache_size: 100 * 1024 * 1024,
})?);
// Create gradient store
let gradient_store = GradientStore::new(store);
// Store a gradient with delta encoding
let gradient = vec![1.0f32, 2.0, 3.0, 4.0];
let metadata = ProvenanceMetadata {
layer: "layer1".to_string(),
timestamp: 1234567890,
training_config: "lr=0.001".to_string(),
};
let cid = gradient_store.store_gradient(&gradient, Some(metadata)).await?;Structs§
- Compression
Stats - Compression statistics
- Delta
Encoder - Delta encoder for efficient gradient storage
- Gradient
Data - Gradient data with metadata
- Gradient
Store - Gradient store for managing gradients with delta encoding
- Provenance
Metadata - Provenance metadata for tracking gradient origins