Struct BlockQuantizedMatrix 

pub struct BlockQuantizedMatrix<'a, T> { /* private fields */ }

Matrix which is quantized into blocks along the K dimension.

The data layout and supported bit/block sizes follow ONNX Runtime’s MatMulNBits operator. See https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md.

Implementations

impl<'a, T: Copy> BlockQuantizedMatrix<'a, T>

pub const MIN_BLOCK_SIZE: usize = 16

Minimum supported number of elements per block.

pub fn new( quant: Contiguous<NdTensorView<'a, u8, 3>>, scales: Contiguous<NdTensorView<'a, T, 2>>, bits: u8, ) -> Result<Self, BlockQuantizedError>

Create a block-quantized RHS matrix input.

quant is the block-quantized input with shape (cols, k_blocks, block_size). scales are the per-block scales with shape (cols, k_blocks). bits specifies the number of bits per element in each block.

The number of elements per quantized block must be a power of 2 of at least MIN_BLOCK_SIZE.

pub fn rows(&self) -> usize

Return the number of rows in the dequantized matrix.

pub fn cols(&self) -> usize

Return the number of columns in the dequantized matrix.

pub fn n_bits(&self) -> u8

Return the number of bits per element

Trait Implementations

impl<'a, T: Clone> Clone for BlockQuantizedMatrix<'a, T>

fn clone(&self) -> BlockQuantizedMatrix<'a, T>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a, T: Copy> Copy for BlockQuantizedMatrix<'a, T>