Skip to main content

GenerativeModel

combs_models

Trait GenerativeModel 

Source 
pub trait GenerativeModel<B: Backend>: Send {
    // Required methods
    fn metadata(&self) -> &ModelMetadata;
    fn load(source: &dyn ModelSource, device: &Device<B>) -> Result<Self>
       where Self: Sized;
    fn create_kv_cache(&self, config: &CacheConfig) -> Box<dyn KVCache<B>>;
    fn embed(&self, tokens: Tensor<B, 2, Int>) -> Tensor<B, 3>;
    fn prefill(
        &mut self,
        input: Tensor<B, 3>,
        cache: &mut dyn KVCache<B>,
        pos: Range<u32>,
    ) -> Tensor<B, 2>;
    fn decode(
        &mut self,
        input: Tensor<B, 3>,
        cache: &mut dyn KVCache<B>,
    ) -> Tensor<B, 2>;

    // Provided method
    fn embed_multimodal(
        &self,
        tokens: Tensor<B, 2, Int>,
        images: &[Tensor<B, 4>],
    ) -> Result<Tensor<B, 3>> { ... }
}
Expand description

Fixed contract every generative architecture implements — the direct analog of MLC’s embed / prefill / decode / create_kv_cache function set. The runtime only ever talks to models through this trait.

Required Methods§

Source

fn metadata(&self) -> &ModelMetadata

Metadata this model was built from.

Source

fn load(source: &dyn ModelSource, device: &Device<B>) -> Result<Self>
where Self: Sized,

Loads all weights from a ModelSource onto device.

Source

fn create_kv_cache(&self, config: &CacheConfig) -> Box<dyn KVCache<B>>

Creates a fresh KV cache for a new generation session, sized and implemented according to config (paged arena vs contiguous baseline).

Source

fn embed(&self, tokens: Tensor<B, 2, Int>) -> Tensor<B, 3>

Embeds token ids: [batch, seq] -> [batch, seq, hidden].

Source

fn prefill( &mut self, input: Tensor<B, 3>, cache: &mut dyn KVCache<B>, pos: Range<u32>, ) -> Tensor<B, 2>

Runs (a chunk of) the prompt through the model, filling the KV cache for positions pos. pos.end - pos.start must equal the input sequence length, and pos.start must equal the cache’s current length (dense contiguous chunks). Returns the logits of the last position, shape [batch, vocab].

Source

fn decode( &mut self, input: Tensor<B, 3>, cache: &mut dyn KVCache<B>, ) -> Tensor<B, 2>

Runs one decode step (single new position at the end of the cache). Returns the logits of that position, shape [batch, vocab].

Provided Methods§

Source

fn embed_multimodal( &self, tokens: Tensor<B, 2, Int>, images: &[Tensor<B, 4>], ) -> Result<Tensor<B, 3>>

Embeds token ids, splicing vision-tower features into the image-token spans. images are preprocessed pixel batches [1, channels, H, W], one per image-token span, in order. Text-only models keep the default impl, which rejects non-empty media and otherwise defers to embed.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§