Struct QuantizedBytes 

pub struct QuantizedBytes {
    pub bytes: Bytes,
    pub scheme: QuantScheme,
    pub shape: Shape,
}

Quantized data bytes representation.

Notes

1. The quantized values are packed into 32-bit unsigned integers. For example, int8 quantized values pack 4 grouped values into a single u32. When unpacking these values, we make sure to retrieve only the meaningful values (and ignore the alignment padding).
2. Quantization parameters are appended to the tensor data. As such, the last bytes always correspond to the scale parameter. If the quantization scheme includes an offset (zero-point) parameter, it is next to last.

Fields

bytes: Bytes

The quantized values and quantization parameters represented as bytes.

scheme: QuantScheme

The quantization scheme.

shape: Shape

The shape of the quantized tensor. The block count, and so the scale count, follows from it per axis: a block that does not span the trailing dimension is not a run of elements.

Implementations

impl QuantizedBytes

pub fn new<E: CheckedBitPattern + NoUninit>( value: Vec<E>, shape: impl Into<Shape>, scheme: QuantScheme, scales: &[f32], global: Option<f32>, ) -> Self

Creates a new quantized bytes representation.

global is the per-tensor scale, required by a two-level scheme and rejected by a one-level one.

pub fn num_elements(&self) -> usize

The number of quantized elements.

pub fn into_vec_i8(self) -> (Vec<i8>, DecodedScales)

Returns the int8 quantized values with the quantization parameters.