Module hash_roll::zstd

zstd's --rsyncable option performs content defined chunking

This has been minimally validated to match the implimentation from zstd, with the following caveats:

• Maximum chunk size is not implimented
• Only 1 test case with a single chunk edge (ie: 2 chunks) has been tested

It uses a internal rolling hash with 1 multiple and 2 additions. (see ZSTD_rollingHash_append() for core functionality).

The rolling hash is then used by findSynchronizationPoint() in various ways to find "syncronization points" (ie: edges of chunks).

This issue thread comment also includes some explanation on the mechanism.

The zstd code does include in it's context information about previous block that was emitted. In other words: the rolling hash isn't "reset" on block emittion. (Most chunking algorithms are reset on block emittion).

Structs

Zstd
ZstdIncr	Incrimental chunking using Zstd's rsyncable algorithm
ZstdSearchState