1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
//! Hand-written SIMD kernels for KevaVolt's hottest inner loops.
//!
//! # Why this crate exists
//!
//! Two claims are often confused, and only one of them is true here.
//!
//! *False*: "assembly makes a key-value store fast." It does not. The order of
//! magnitude comes from running commands on every core instead of one, from
//! `io_uring`, and from a memory layout that answers a probe out of a single
//! cache line. Those are architectural, and no amount of assembly substitutes
//! for them.
//!
//! *True*: once the working set is cache-resident, instruction count **is** the
//! runtime, and compilers leave measurable performance on the table there.
//! FFmpeg's own assembly documentation puts intrinsics at 10-15% behind
//! hand-written assembly, and dav1d measured 2x from auto-vectorisation against
//! 8x from hand-written kernels. The cost is real: register allocation and
//! instruction scheduling stay with the compiler when you use intrinsics, and
//! it will happily hoist a load out of a loop in a way that costs you a port.
//!
//! *Also true, and measured here*: that advantage is a function of kernel size,
//! not of assembly as such. Below roughly a dozen instructions, the fact that an
//! `extern "C"` kernel cannot be inlined costs more than better scheduling
//! returns. Our own control-group probe demonstrates it -- the hand-written NEON
//! version loses to the intrinsics by 47% purely on `call`/`ret`.
//!
//! So this crate is deliberately small, and a kernel earns the production path
//! only by beating its **intrinsics** version in `benches/` -- beating the
//! scalar loop merely shows that SIMD beats no SIMD. Assembly is reserved for
//! kernels long enough to amortise the call boundary: the expiry sweep,
//! eviction sampling, defragmentation. See
//! `docs/DESIGN_DECISION_ASM_KERNELS.md` for the admission criteria.
//!
//! # Correctness
//!
//! Every kernel has a portable reference in `format`, reachable by building
//! without the `asm` feature, and the differential tests assert that the two
//! agree. The portable path is a supported configuration, not a
//! fallback of last resort: building with `--no-default-features` disables the
//! assembly entirely and must still pass the full test suite.
// The group probe that used to live here -- `ctrl_match`, `BitMask`, their
// scalar and intrinsic backends, and the group width they were built around --
// stayed behind in the store it belongs to. It shared no symbol, no constant
// and no file with the codec, which is what made this split a matter of moving
// files rather than untangling them.
/// Finding matches: the hash table walk and the assembled packers.
/// Reading a stream back: the dispatch across split, format and part line.
pub const
/// Human-readable name of the selected backend, for labelling a measurement.
///
/// A benchmark number that cannot be traced back to the code path that produced
/// it is worthless, so this is reported alongside every result.
///
/// It therefore has to report the one distinction that changes the number by a
/// factor rather than a percent: whether the assembled kernels are in or out.
/// It used to name only the architecture's SIMD flavour, so a run with `asm`
/// disabled and a run with it enabled both printed `neon-intrinsics` -- which
/// is precisely the confusion the function exists to prevent.
///
/// Which part line runs inside the assembly is decided at runtime by
/// [`crate::raw`]'s dispatch and cannot be named from a `const fn`.
pub const