Struct llm_base::InferenceSnapshot

pub struct InferenceSnapshot {
    pub npast: usize,
    pub session_params: InferenceSessionParameters,
    pub tokens: Vec<TokenId>,
    pub last_logits: Vec<f32>,
    pub memory_k: Vec<u8>,
    pub memory_v: Vec<u8>,
}

A serializable snapshot of the inference process. Can be restored by calling InferenceSession::from_snapshot.

Fields

npast: usize

How many tokens have been stored in the memory so far.

session_params: InferenceSessionParameters

Parameters associated with the saved inference session.

tokens: Vec<TokenId>

All tokens generated by this inference session.

last_logits: Vec<f32>

The vector of logits that was produced after the last inference.

memory_k: Vec<u8>

The contents of the ‘key’ memory tensor.

memory_v: Vec<u8>

The contents of the ‘value’ memory tensor.

Trait Implementations

impl Clone for InferenceSnapshot

fn clone(&self) -> InferenceSnapshot

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'de> Deserialize<'de> for InferenceSnapshot

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq<InferenceSnapshot> for InferenceSnapshot

fn eq(&self, other: &InferenceSnapshot) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for InferenceSnapshot