Struct BlockDeduplicator 

pub struct BlockDeduplicator { /* private fields */ }

Content-aware block deduplicator using variable-length CDC chunking.

Stores objects by splitting them into content-defined chunks, using the chunk hash as the storage key. Identical chunks are stored only once and reference-counted. When all objects referencing a chunk are deleted, the chunk data is freed.

Algorithm

Content-Defined Chunking (CDC) uses a rolling hash over a sliding window:

• For each byte position after min_chunk_size into the current chunk, compute rolling_hash(data[pos-W..pos]) where W = window_size.
• If hash & mask == 0 (where mask = 2^target_bits - 1), emit a boundary.
• Also emit a boundary when chunk_length >= max_chunk_size.
• The final remaining bytes are always emitted as the last chunk.

Thread safety

BlockDeduplicator is not Sync. Use external locking (e.g. Mutex) when sharing across threads.

Implementations

impl BlockDeduplicator

pub fn new(config: ChunkingConfig) -> Self

Create a new BlockDeduplicator with the given configuration.

pub fn with_defaults() -> Self

Create a BlockDeduplicator with default configuration.

pub fn chunk_data(&self, data: &[u8]) -> Vec<(ChunkHash, Vec<u8>)>

Split data into content-defined chunks using the rolling-hash CDC algorithm.

Returns an ordered list of (hash, chunk_bytes) pairs. The concatenation of all chunk_bytes is guaranteed to equal data exactly.

Edge cases

• Empty input → returns an empty Vec.
• Input shorter than min_chunk_size → returns a single chunk.
• Input shorter than window_size → rolling hash uses the full available prefix.

pub fn store_object( &mut self, object_id: String, data: Vec<u8>, ) -> Result<ObjectManifest, DeduplicatorError>

Store an object, deduplicating its content-defined chunks.

If an identical chunk (same hash) already exists, its ref_count is incremented and the duplicate bytes are not stored again. A manifest is built and stored internally.

Errors

Currently infallible, but returns Result for future extensibility (e.g., when storage-full limits are enforced).

pub fn retrieve_object( &self, object_id: &str, ) -> Result<Vec<u8>, DeduplicatorError>

Retrieve the full byte content of a stored object by reconstructing it from its chunk manifest.

Errors

• DeduplicatorError::ObjectNotFound if object_id has no manifest.
• DeduplicatorError::ChunkNotFound if a chunk referenced by the manifest is missing from the store (indicates data corruption).
• DeduplicatorError::InvalidManifest if the reconstructed size does not match the manifest’s declared total_size.

pub fn delete_object( &mut self, object_id: &str, ) -> Result<Vec<ChunkHash>, DeduplicatorError>

Delete a stored object, decrementing ref counts of its chunks.

Any chunk whose ref_count reaches zero is removed from the store. Returns the hashes of all physically removed chunks.

Errors

• DeduplicatorError::ObjectNotFound if object_id has no manifest.

pub fn get_chunk(&self, hash: &ChunkHash) -> Result<&Chunk, DeduplicatorError>

Return a reference to the Chunk metadata for the given hash.

Errors

• DeduplicatorError::ChunkNotFound if no chunk with that hash exists.

pub fn chunk_exists(&self, hash: &ChunkHash) -> bool

Return true if a chunk with the given hash is present in the store.

pub fn compact(&mut self) -> usize

Remove all chunks whose ref_count == 0 from the store.

This is a garbage-collection pass for chunks that were decremented to zero during earlier delete_object calls but not yet physically removed (e.g., if delete_object was called but removal was deferred). In the current implementation, delete_object removes zero-ref chunks eagerly, so this acts as a safety pass.

Returns the number of chunks removed.

pub fn stats(&self) -> DeduplicationStats

Compute and return a snapshot of current deduplication statistics.

pub fn list_objects(&self) -> Vec<String>

Return a sorted list of all stored object IDs.

pub fn chunk_count(&self) -> usize

Return the number of unique chunks currently in the store.

pub fn object_count(&self) -> usize

Return the number of objects currently stored.

pub fn get_manifest(&self, object_id: &str) -> Option<&ObjectManifest>

Retrieve the manifest for a stored object without reconstructing data.

pub fn config(&self) -> &ChunkingConfig

Return a reference to the current chunking configuration.