Struct LlamaModelV2 

pub struct LlamaModelV2 { /* private fields */ }

Main Llama model implementation

Implementations

impl LlamaModelV2

pub fn forward_with_cache( &self, inputs: &ModelInputs, cache: Option<&mut KVCache>, ) -> Result<ModelOutputs>

Forward pass with KV cache support

This method enables efficient autoregressive generation by caching Key and Value tensors from previous tokens.

Arguments

• inputs - Model inputs (only new tokens)
• cache - Optional mutable reference to KV cache

Returns

Model outputs with logits for the new tokens

pub fn generate_with_cache( &self, prompt: &str, config: &GenerationConfig, ) -> Result<String>

Generate text with KV caching for efficient autoregressive generation

This method is significantly faster than generate() for long sequences because it caches Key and Value tensors instead of recomputing them.

Arguments

• prompt - Input text prompt
• config - Generation configuration

Returns

Generated text including the prompt

Trait Implementations

impl Model for LlamaModelV2

type Config = LlamaConfig

Model configuration type

fn new(config: Self::Config) -> Result<Self>

Create new model instance

fn from_weights(config: Self::Config, weights: ModelWeights) -> Result<Self>

Load model with weights

fn forward(&self, inputs: &ModelInputs) -> Result<ModelOutputs>

Forward pass - core inference method

fn generate(&self, prompt: &str, config: &GenerationConfig) -> Result<String>

Generate text (high-level interface)

fn config(&self) -> &Self::Config

Get model configuration

fn memory_requirements(&self) -> MemoryRequirements

Get model memory requirements

fn to_device(&mut self, device: &Device) -> Result<()>

Move model to device