Struct LlamaModel 

pub struct LlamaModel<B: Backend> { /* private fields */ }

Llama-family causal LM, parameterized by the resolved ArchSpec — llama, smollm2, qwen2, and mistral today; the gemma/qwen3/phi presets migrate onto it stage by stage (roadmap wave 2).

Trait Implementations

impl<B: Backend> GenerativeModel<B> for LlamaModel<B>

fn metadata(&self) -> &ModelMetadata

Metadata this model was built from.

fn load(source: &dyn ModelSource, device: &Device<B>) -> Result<Self>

Loads all weights from a ModelSource onto device.

fn create_kv_cache(&self, config: &CacheConfig) -> Box<dyn KVCache<B>>

Creates a fresh KV cache for a new generation session, sized and implemented according to config (paged arena vs contiguous baseline).

fn embed(&self, tokens: Tensor<B, 2, Int>) -> Tensor<B, 3>

Embeds token ids: [batch, seq] -> [batch, seq, hidden].

fn prefill( &mut self, input: Tensor<B, 3>, cache: &mut dyn KVCache<B>, pos: Range<u32>, ) -> Tensor<B, 2>

Runs (a chunk of) the prompt through the model, filling the KV cache for positions pos. pos.end - pos.start must equal the input sequence length, and pos.start must equal the cache’s current length (dense contiguous chunks). Returns the logits of the last position, shape [batch, vocab].

fn prefill_hidden( &mut self, input: Tensor<B, 3>, cache: &mut dyn KVCache<B>, pos: Range<u32>, ) -> Result<Tensor<B, 3>>

Runs (a chunk of) the prompt and returns the final-norm hidden states for those positions, shape [1, seq, hidden] — the embeddings path. Same cache/position contract as GenerativeModel::prefill. Models that cannot expose hidden states keep the default error.

fn supports_hidden_states(&self) -> bool

Whether GenerativeModel::prefill_hidden is implemented — the capability flag /v1/model/info advertises as embeddings.

fn prefill_all_logits( &mut self, input: Tensor<B, 3>, cache: &mut dyn KVCache<B>, pos: Range<u32>, ) -> Result<Tensor<B, 3>>

Runs (a chunk of) the prompt and returns logits for every position, shape [1, seq, vocab] — the perplexity / speculative- decode path. Same cache/position contract as GenerativeModel::prefill. Memory scales with seq × vocab, so callers chunk accordingly. Default: unsupported.

fn decode( &mut self, input: Tensor<B, 3>, cache: &mut dyn KVCache<B>, ) -> Tensor<B, 2>

Runs one decode step (single new position at the end of the cache). Returns the logits of that position, shape [batch, vocab].

fn decode_all_logits( &mut self, input: Tensor<B, 3>, cache: &mut dyn KVCache<B>, ) -> Result<Tensor<B, 3>>

Decodes n tokens at the cache tail and returns logits for every position ([1, n, vocab]), not just the last row — the seam multi-token verification needs. Architectures without it never take the speculative path.

fn supports_decode_all_logits(&self) -> bool

Whether GenerativeModel::decode_all_logits is implemented.

fn embed_multimodal( &self, tokens: Tensor<B, 2, Int>, images: &[Tensor<B, 4>], ) -> Result<Tensor<B, 3>>

Embeds token ids, splicing vision-tower features into the image-token spans. images are preprocessed pixel batches [1, channels, H, W], one per image-token span, in order. Text-only models keep the default impl, which rejects non-empty media and otherwise defers to embed.