Expand description
zstd’s --rsyncable option performs content defined chunking
This has been minimally validated to match the implimentation from zstd, with the following caveats:
- Maximum chunk size is not implimented
- Only 1 test case with a single chunk edge (ie: 2 chunks) has been tested
It uses a internal rolling
hash
with 1 multiple and 2 additions. (see ZSTD_rollingHash_append() for core functionality).
The rolling hash is then used by
findSynchronizationPoint()
in various ways to find “syncronization points” (ie: edges of chunks).
This issue thread comment also includes some explanation on the mechanism.
The zstd code does include in it’s context information about previous block that was emitted. In other words: the rolling hash isn’t “reset” on block emittion. (Most chunking algorithms are reset on block emittion).
Structs§
- Zstd
- Zstd
Incr - Incrimental chunking using Zstd’s rsyncable algorithm
- Zstd
Search State