Crate mothcdc 

MothCDC

A fork of Orson Peters’ MinCDC that adds a caterpillar coalescing layer for metadata-efficient content-defined chunking on redundant data.

The core chunking algorithm lives in mincdc (inherited from upstream MinCDC); the coalescing layer lives in caterpillar, built on top of it.

The default MothChunker and MothReadChunker use the robust hashed MinCDC boundary selector and emit validated Segment values. A segment is either one ordinary chunk or a maximal run of identical adjacent chunks; use Segment::dedup_key for storage and Segment::len for its logical byte length.

In-memory example

use mothcdc::MothChunker;

let data = vec![0u8; 64 * 4096]; // a long zero run
let segs: Vec<_> = MothChunker::new(&data, 4096, 12288).collect();

// The whole zero run collapses into one record instead of ~64 chunks.
assert_eq!(segs.len(), 1);
assert!(segs[0].is_caterpillar());
assert_eq!(segs[0].len(), data.len() as u64);

Streaming example

use std::io::Cursor;
use mothcdc::MothReadChunker;

let data = vec![0u8; 128 * 4096];
let mut chunker = MothReadChunker::new(Cursor::new(&data), 4096, 12288);
let mut restored = Vec::new();
while let Some(segment) = chunker.next()? {
    // Streaming segments borrow the internal buffer until the next call.
    segment.reconstruct_into(&mut restored);
}
assert_eq!(restored, data);

Re-exports

pub use mincdc::Chunk;

pub use caterpillar::MothChunker;

pub use caterpillar::MothReadChunker;

pub use caterpillar::Segment;

Modules

caterpillar

Caterpillar coalescing layer: metadata efficiency on redundant data. Caterpillar coalescing — a metadata-efficiency layer over the chunkers.

mincdc

The core MinCDC chunking algorithm, inherited from Orson Peters’ MinCDC: mincdc::Cdc, mincdc::MinCdc4, mincdc::MinCdcHash4, mincdc::SliceChunker, mincdc::ReadChunker, and mincdc::Chunk. MinCDC