pub struct QuantizedLinearModel {
pub name: String,
pub vocab_size: usize,
pub hidden_dim: usize,
pub quant_type: QuantType,
pub quantized_weights: Vec<u8>,
pub bytes_per_row: usize,
pub embeddings: Vec<f32>,
}Expand description
A memory-efficient quantized linear projection model (Q8_0 or Q4_0). Reduces LM head weight memory by up to 75%–87% compared to FP32.
Fields§
§name: String§vocab_size: usize§quant_type: QuantType§quantized_weights: Vec<u8>Raw quantized bytes matrix of shape [vocab_size, bytes_per_row].
bytes_per_row: usize§embeddings: Vec<f32>Unquantized or dequantized token embedding table.
Implementations§
Source§impl QuantizedLinearModel
impl QuantizedLinearModel
Sourcepub fn from_f32_weights(
name: impl Into<String>,
vocab_size: usize,
hidden_dim: usize,
weights: &[f32],
embeddings: Vec<f32>,
) -> Result<QuantizedLinearModel, EngineError>
pub fn from_f32_weights( name: impl Into<String>, vocab_size: usize, hidden_dim: usize, weights: &[f32], embeddings: Vec<f32>, ) -> Result<QuantizedLinearModel, EngineError>
Creates a Q8_0 quantized linear model from FP32 weights.
Projects hidden states into logits using quantized dot products.
Sourcepub fn project_token(&self, token: u32) -> Logits
pub fn project_token(&self, token: u32) -> Logits
Computes logits directly for an input token ID.
Trait Implementations§
Source§impl Clone for QuantizedLinearModel
impl Clone for QuantizedLinearModel
Source§fn clone(&self) -> QuantizedLinearModel
fn clone(&self) -> QuantizedLinearModel
Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source. Read moreSource§impl Debug for QuantizedLinearModel
impl Debug for QuantizedLinearModel
Source§impl ModelEngine for QuantizedLinearModel
impl ModelEngine for QuantizedLinearModel
Source§async fn forward_batch(
&mut self,
tokens: &[u32],
) -> Result<Vec<Logits>, EngineError>
async fn forward_batch( &mut self, tokens: &[u32], ) -> Result<Vec<Logits>, EngineError>
Computes logits for a sequence of tokens in a single parallel forward pass.
Source§async fn forward_token(&mut self, token: u32) -> Result<Logits, EngineError>
async fn forward_token(&mut self, token: u32) -> Result<Logits, EngineError>
Computes logits for a single token step.
Auto Trait Implementations§
impl Forget for QuantizedLinearModel
impl Freeze for QuantizedLinearModel
impl Move for QuantizedLinearModel
impl RefUnwindSafe for QuantizedLinearModel
impl Send for QuantizedLinearModel
impl Sync for QuantizedLinearModel
impl Unpin for QuantizedLinearModel
impl UnsafeUnpin for QuantizedLinearModel
impl UnwindSafe for QuantizedLinearModel
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more