Crate gearhash

Source
Expand description

The GEAR hashing function is a fast, rolling hash function that is well suited for content defined chunking. In particular, it is used as a building block for the FastCDC algorithm.

The implementation provided in this crate consists of both a simple, scalar variant, as well as optimized versions for the SSE4.2 and AVX2 instruction sets.

§Example

fn find_all_chunks(buf: &[u8], mask: u64) -> Vec<&[u8]> {
    // set up initial state
    let mut chunks = vec![];
    let mut offset = 0;

    // create new hasher
    let mut hasher = gearhash::Hasher::default();

    // loop through all matches, and push the corresponding chunks
    while let Some(boundary) = hasher.next_match(&buf[offset..], mask) {
        chunks.push(&buf[offset..offset + boundary]);
        offset += boundary;
    }

    // push final chunk
    chunks.push(&buf[offset..]);
    chunks
}

Structs§

  • Hasher
    Gear hash state. Processes bytes to find chunk boundaries.

Statics§

  • DEFAULT_TABLE
    Default hash table, using random (but static) integers.

Type Aliases§