Expand description
Count-min frequency sketch keyed directly on BLAKE3 hashes (ADR 040).
Keys are already uniform 32-byte hashes, so each row’s column index is a
disjoint 4-byte slice of the key reduced mod the sketch’s width — no hash
functions. Counters are saturating u8 — classic TinyLFU uses 4-bit
counters, and the extra headroom is free at one byte each. Halving ages them.
ShardedCountMinSketch is the form the estimator uses. It splits the
counter array into SHARDS independently locked CountMinSketches of
cols / SHARDS columns each, routed by a 4-byte slice of the key that no
row index reads. An increment or an estimate locks exactly one shard.
Two properties of one wide sketch of cols columns are load-bearing for the
policies that read the estimates, and sharding keeps both:
- Aging cadence is global, not per shard. Both readers compare estimates
against a shared scale:
super::tinylfu::ProbationAdmissiontests one estimate against a node-widepromotion_threshold, andsuper::tinylfu::TinyLfuEvictionranks candidates from different shards least-frequent-first. Counters are only comparable if every counter is caught up to the same clock when it is read. So the halving clock is one node-wide observation counter, and a shard halves lazily — on its next touch — by however many windows have elapsed since it last caught up. The work stays sharded (1 / SHARDSof the array under one shard’s lock, landing on one observer), while the cadence stays global. A shard clocked on its own traffic would age at a rate set by key skew, which is exactly what a frequency sketch must not do. - Per-row collision rate. Two distinct keys share a row counter only if
they share a shard (probability
1 / SHARDS) and then a column within it (SHARDS / cols) — the same1 / colsa single wide sketch gives, because the shard slice and the row slices are disjoint bytes of a uniform hash.
The over-report rate carries over as exactly as the per-row rate does. The
sketch reads a key hotter than it is when every one of its ROWS counters
also holds some other key’s count. The polluting keys need not be the same
one across rows, so “two keys collide in all ROWS rows” is a far rarer
event that does not bound the error — do not size the sketch from it. For
N live keys the rate is (1 - e^(-N / cols))^ROWS, and SHARDS cancels
out: a shard divides the columns and the keys in the same proportion. So
sharding costs nothing in accuracy and the width alone sets it. The rate
turns on N / cols, which holds the shipped cols = 65536 under one
percent out to roughly 25000 live keys. See
super::tinylfu::TinyLfuEstimator::new for the sizing and its floor.
Structs§
- Count
MinSketch - A plain count-min counter array. It carries no aging clock of its own:
ShardedCountMinSketchowns the node-wide clock and callsSelf::halve, which is what keeps every shard on one cadence. - Sharded
Count MinSketch - A
CountMinSketchsplit intoSHARDSindependently locked pieces on one node-wide aging clock.
Constants§
- SHARDS
- Number of independently locked shards in a
ShardedCountMinSketch.