Trait CacheTarget 

pub trait CacheTarget: Debug {
    // Required methods
    fn num_layers(&self) -> u32;
    fn append_key(
        &mut self,
        layer: u32,
        position: u32,
        key: &[f32],
    ) -> Result<()>;
    fn append_value(
        &mut self,
        layer: u32,
        position: u32,
        value: &[f32],
    ) -> Result<()>;
    fn seq_len(&self) -> u32;
}

Trait for KV cache targets that can receive injected blocks.

Required Methods

fn num_layers(&self) -> u32

Get the number of layers in this cache.

fn append_key(&mut self, layer: u32, position: u32, key: &[f32]) -> Result<()>

Append a key block at a specific layer and position.

fn append_value( &mut self, layer: u32, position: u32, value: &[f32], ) -> Result<()>

Append a value block at a specific layer and position.

fn seq_len(&self) -> u32

Get the current sequence length (tokens in cache).

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors