Skip to main content

Module chunking

Module chunking 

Source

Structs§

Blake3Hasher
BLAKE3 hasher
Chunk
Represents a chunk with its metadata
FastCDCChunker
FastCDC-based chunking (content-defined)
FixedChunker
Fixed-size chunking (for comparison/benchmarking)
PipelineConfig

Enums§

ChunkEvent
Result of chunking operation

Constants§

CHUNK_AVG
FastCDC average target of the default configuration.
CHUNK_MAX
FastCDC upper bound of the default configuration.
CHUNK_MIN
FastCDC lower bound of the default configuration.
CHUNK_ROOT_PREFIX
The root hash of a file under ffsync’s chunk-tree definition: BLAKE3(CHUNK_ROOT_PREFIX || chunk_hashes_in_file_order) where CHUNK_ROOT_PREFIX is the 32-byte personalization below. This is NOT the standard BLAKE3 file hash (whose internal tree is over fixed 1024-byte blocks, not CDC chunk boundaries); it is a single BLAKE3 hash over the ordered chunk-hash list, domain-separated from every other BLAKE3 use in this codebase.
SMALL_CHUNK_AVG
Small-file average target.
SMALL_CHUNK_MAX
Small-file upper bound.
SMALL_CHUNK_MIN
Small-file configuration (D19): the F1-sweep best point.
SMALL_FILE_BOUNDARY
Files below this size chunk with the small-file configuration (D19). The size-ladder sweep (evaluation/chunking.md, 2026-10-23) puts the crossover here: below 16 MiB the smaller config wins unambiguously (rust-src 8 MiB: 18.1% → 4.1% delta; tos.webm 16 MiB: 8.6% → 3.8%), at 32 MiB the difference is noise, and chunk counts stay trivial at the boundary (~200 chunks ≈ 9 KiB manifest/index per file).

Traits§

Chunker
Trait for different chunking strategies
Hasher
Trait for different hashing strategies

Functions§

cdc_params_for_size
The CDC parameters for a file of size bytes (D19): a pure function of the size, evaluated by the sender before scanning and sent in HELLO (cdc_min/cdc_avg/cdc_max) — the receiver honors the wire values, so this policy only decides what the sender sends. Files at or above the boundary chunk exactly as the shipped defaults always have, which keeps every existing large-file index valid.
chunk_root
Computes the chunk-tree root hash from the ordered chunk list of a file.
process_file
sane_cdc_params
Sanity-checks peer-supplied CDC parameters (HELLO): usable only when ordered and non-degenerate. Nonsense values fall back to fallback instead of trusting the wire — a hostile peer must not be able to steer our chunker into a pathological configuration.
scan_file_ordered
Scans a file through the pipeline and returns all chunks in canonical file order (sorted by offset).