Module gradient

Module gradient 

Source
Expand description

Gradient and tensor storage with delta encoding

Provides efficient storage for neural network gradients and tensors:

  • Delta encoding (store changes only)
  • Sparse gradient compression
  • Provenance metadata tracking
  • Integration with version control

§Example

use ipfrs_storage::{GradientStore, DeltaEncoder, SledBlockStore, BlockStoreConfig};
use std::sync::Arc;
use std::path::PathBuf;

// Create block store
let store = Arc::new(SledBlockStore::new(BlockStoreConfig {
    path: PathBuf::from(".ipfrs/gradients"),
    cache_size: 100 * 1024 * 1024,
})?);

// Create gradient store
let gradient_store = GradientStore::new(store);

// Store a gradient with delta encoding
let gradient = vec![1.0f32, 2.0, 3.0, 4.0];
let metadata = ProvenanceMetadata {
    layer: "layer1".to_string(),
    timestamp: 1234567890,
    training_config: "lr=0.001".to_string(),
};

let cid = gradient_store.store_gradient(&gradient, Some(metadata)).await?;

Structs§

CompressionStats
Compression statistics
DeltaEncoder
Delta encoder for efficient gradient storage
GradientData
Gradient data with metadata
GradientStore
Gradient store for managing gradients with delta encoding
ProvenanceMetadata
Provenance metadata for tracking gradient origins