firecloud-storage

Chunking, compression, and local storage for the FireCloud distributed storage system.

Overview

Handles file chunking, compression, and persistent storage:

• Content-defined chunking: FastCDC algorithm for deduplication
• Compression: Zstd and LZ4 support
• Storage: Embedded Sled database
• Manifests: File metadata and chunk tracking

Usage

use firecloud_storage::{ChunkStore, Chunker, CompressionType};

// Open storage
let store = ChunkStore::open("./data")?;

// Chunk and store a file
let chunker = Chunker::new(file_data, CompressionType::Zstd);
for chunk in chunker {
    store.put_chunk(&chunk)?;
}

Features

• FastCDC: Content-defined chunking for optimal deduplication
• Multiple compression algorithms: Zstd (default) and LZ4
• Embedded database: No external dependencies with Sled
• Async support: Built on Tokio

Performance

• Chunk sizes: 256KB average, 64KB-1MB range
• Compression: ~60% reduction with Zstd
• Deduplication: Automatic with content-defined chunking

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.