1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
//! Block layouts mirroring ggml's `ggml-common.h`, plus the activation-side
//! block structs.
//!
//! Weight blocks are described as byte offsets into the packed on-disk form
//! (all fields little-endian, no padding -- the C structs are naturally
//! packed). Dequantization formulas, with `f16()` meaning f16-to-f32:
//!
//! - `block_q8_0` (34 B / 32 weights): `d: f16` at 0, `qs: [i8; 32]` at 2.
//! `w[j] = f16(d) * qs[j]`.
//! - `block_q4_K` (144 B / 256 weights): `d: f16` at 0, `dmin: f16` at 2,
//! `scales: [u8; 12]` at 4 (eight 6-bit (scale, min) pairs, packed per
//! [`get_scale_min_k4`]), `qs: [u8; 128]` at 16. Eight sub-blocks of 32.
//! For sub-block `j` with unpacked `(sc, m)`:
//! `w = f16(d)*sc*q - f16(dmin)*m`, `q` a 4-bit unsigned value. Nibble
//! order: within each 64-weight span, `qs` byte `l` holds weight `l` in
//! its low nibble and weight `l + 32` in its high nibble.
//! - `block_q5_K` (176 B / 256 weights): `d: f16` at 0, `dmin: f16` at 2,
//! `scales: [u8; 12]` at 4 (same packing as q4_K), `qh: [u8; 32]` at 16,
//! `qs: [u8; 128]` at 48. Same affine formula as q4_K with a 5-bit `q`:
//! the low 4 bits come from `qs` (q4_K nibble order) and the 5th bit from
//! `qh` -- for the 64-weight span `s` (0..4), weight `l` of the low-nibble
//! half uses `qh[l] & (1 << 2s)` and weight `l` of the high-nibble half
//! uses `qh[l] & (2 << 2s)`.
//! - `block_q6_K` (210 B / 256 weights): `ql: [u8; 128]` at 0,
//! `qh: [u8; 64]` at 128, `scales: [i8; 16]` at 192, `d: f16` at 208.
//! Sixteen sub-blocks of 16: `w = f16(d) * scales[i/16] * q` with
//! `q = ((low 4 bits) | (2 bits from qh) << 4) - 32` in [-32, 31]. Within
//! each 128-weight half (using half-local `ql`/`qh`/`scales` slices and
//! `l` in 0..32): weight `l` is `ql[l] & 0xF` with `qh[l] & 3`; weight
//! `l + 32` is `ql[l + 32] & 0xF` with `(qh[l] >> 2) & 3`; weight `l + 64`
//! is `ql[l] >> 4` with `(qh[l] >> 4) & 3`; weight `l + 96` is
//! `ql[l + 32] >> 4` with `(qh[l] >> 6) & 3`.
//! - `block_q8_K` (292 B / 256 weights, activation-only here): `d: f32`,
//! `qs: [i8; 256]`, `bsums: [i16; 16]` -- `bsums[g]` is the sum of the
//! 16-element group `qs[16g .. 16g + 16]`, which lets the k-quant dots
//! apply the per-sub-block mins without touching `qs` again.
//! `w[j] = d * qs[j]`. Never packed to bytes in ramvamp; see [`BlockQ8K`].
use f16_to_f32;
/// Weights per k-quant super-block (ggml `QK_K`).
pub const QK_K: usize = 256;
/// Weights per legacy q8_0 block (ggml `QK8_0`).
pub const QK8_0: usize = 32;
/// Byte offsets inside a packed `block_q4_K`.
pub
/// Byte offsets inside a packed `block_q5_K`.
pub
/// Byte offsets inside a packed `block_q6_K`.
pub
/// Byte offsets inside a packed `block_q8_0`.
pub
/// Read the little-endian f16 at `off` in a block and widen to f32.
///
/// Byte-wise load: no alignment assumed beyond 1 byte.
pub
/// Borrow the 12-byte k-quant scale field out of a block.
///
/// Infallible for callers that pass in-bounds constant offsets into a
/// length-validated block; `expect` documents the invariant.
pub
/// Unpack the 6-bit (scale, min) pair for sub-block `j` (0..8) from the
/// 12-byte k-quant scale field.
///
/// Mirrors ggml's `get_scale_min_k4`: pairs 0..4 use the low 6 bits of
/// bytes `j` / `j + 4`; pairs 4..8 use the low nibbles of bytes `j + 4` as
/// low bits and the previously unused high 2 bits of bytes `j - 4` / `j` as
/// high bits.
pub
/// One Q8_K activation block: 256 int8 weights with an f32 scale and
/// per-16-element group sums.
///
/// In-memory analogue of ggml's `block_q8_K`. Produced per token by
/// [`super::quantize_row_q8_k`]; never serialized. `bsums[g]` must equal
/// `qs[16g..16g + 16].iter().sum()` -- the k-quant dot products rely on it
/// to fold the per-sub-block mins in via one multiply per group.
/// One Q8_0 activation block: 32 int8 weights with a single scale.
///
/// In-memory analogue of ggml's `block_q8_0`, except `d` is kept as f32.
/// [`super::quantize_row_q8_0`] stores the f16-rounded value (the f32 you
/// get from reading back the f16 ggml would have written), so arithmetic
/// against it matches llama.cpp's `vec_dot_q8_0_q8_0` exactly.