firecloud-storage 0.2.0

Chunking, compression, and local storage for FireCloud distributed storage
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

# firecloud-storage

Chunking, compression, and local storage for the FireCloud distributed storage system.

## Overview

Handles file chunking, compression, and persistent storage:

- **Content-defined chunking**: FastCDC algorithm for deduplication
- **Compression**: Zstd and LZ4 support
- **Storage**: Embedded Sled database
- **Manifests**: File metadata and chunk tracking

## Usage

```rust
use firecloud_storage::{ChunkStore, Chunker, CompressionType};

// Open storage
let store = ChunkStore::open("./data")?;

// Chunk and store a file
let chunker = Chunker::new(file_data, CompressionType::Zstd);
for chunk in chunker {
    store.put_chunk(&chunk)?;
}
```

## Features

- **FastCDC**: Content-defined chunking for optimal deduplication
- **Multiple compression algorithms**: Zstd (default) and LZ4
- **Embedded database**: No external dependencies with Sled
- **Async support**: Built on Tokio

## Performance

- Chunk sizes: 256KB average, 64KB-1MB range
- Compression: ~60% reduction with Zstd
- Deduplication: Automatic with content-defined chunking

## License

Licensed under either of:

- Apache License, Version 2.0 ([LICENSE-APACHE](../../LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license ([LICENSE-MIT](../../LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.