Skip to main content

quantize_int8

Function quantize_int8 

Source
pub fn quantize_int8(
    v: &[f32],
    out: &mut [i8],
) -> Result<(f32, f32), QuantError>
Expand description

Quantizes one finite row to signed bytes using a per-vector affine map.

A non-constant row reconstructs coordinate i as scale * out[i] + offset, where scale = (max - min) / 254. A constant row uses scale = 0, zero codes, and the constant as its offset, so the edge case is exact. Positive and negative zero are accepted and reconstruct to numerical zero.

ยงErrors

Returns QuantError::EmptyVector for an empty input, QuantError::DimensionTooLarge above the signed-byte kernel limit, QuantError::NonFinite for the first NaN or infinity, and QuantError::OutputLength unless out.len() == v.len(). Validation completes before out is modified.