Skip to main content

hash_chunks

Function hash_chunks 

Source
pub fn hash_chunks(model: &str, tokens: &[u32]) -> Vec<[u8; 32]>
Expand description

Walk a full token sequence and yield one digest per TOKENS_PER_CHUNK rolling chunk. The last (partial) chunk is also emitted because it is often the most cache-relevant.

Position-independent: equal token-windows at different positions hash to the same digest. Use hash_seq_chunks when KV-cache reuse requires positional fidelity.