Crate quickcdc

source ·
Expand description


quickcdc is a fast content defined chunker.

  • For some background information, see AE: An Asymmetric Extremum Content Defined Chunking Algorithm by Yucheng Zhang.
  • Modification(s):
    • User may provide salt, introducing entropy / cutpoint variation (i.e. files re-processed with different salt values will produce different cutpoints).
    • Warp forward (reduced window size), skipping some unnecessary processing that happens before minimum chunk size is reached.

This should be faster than many CDC algorithms (anecdotal performance: 2GB/s on an amd1950x with an NVMe drive), but faster alternatives exist.

  • For more information, see FastCDC

NOTE: This implementation performs much faster when built with --release.