pub struct InferenceSnapshot {
pub npast: usize,
pub config: InferenceSessionConfig,
pub tokens: Vec<TokenId>,
pub last_logits: Vec<f32>,
pub memory_k: Vec<u8>,
pub memory_v: Vec<u8>,
}
Expand description
A serializable snapshot of the inference process. Can be restored by calling InferenceSession::from_snapshot.
Fields§
§npast: usize
How many tokens have been stored in the memory so far.
config: InferenceSessionConfig
Parameters associated with the saved inference session.
tokens: Vec<TokenId>
All tokens generated by this inference session.
last_logits: Vec<f32>
The vector of logits that was produced after the last inference.
memory_k: Vec<u8>
The contents of the ‘key’ memory tensor.
memory_v: Vec<u8>
The contents of the ‘value’ memory tensor.
Trait Implementations§
Source§impl Clone for InferenceSnapshot
impl Clone for InferenceSnapshot
Source§fn clone(&self) -> InferenceSnapshot
fn clone(&self) -> InferenceSnapshot
Returns a duplicate of the value. Read more
1.0.0 · Source§const fn clone_from(&mut self, source: &Self)
const fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source
. Read moreSource§impl<'de> Deserialize<'de> for InferenceSnapshot
impl<'de> Deserialize<'de> for InferenceSnapshot
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
Source§impl PartialEq for InferenceSnapshot
impl PartialEq for InferenceSnapshot
impl StructuralPartialEq for InferenceSnapshot
Auto Trait Implementations§
impl Freeze for InferenceSnapshot
impl RefUnwindSafe for InferenceSnapshot
impl Send for InferenceSnapshot
impl Sync for InferenceSnapshot
impl Unpin for InferenceSnapshot
impl UnwindSafe for InferenceSnapshot
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more