Crate combs_models 

combs-models

Model architecture registry. The runtime drives any architecture through the fixed GenerativeModel contract (MLC’s embed/prefill/decode/ create_kv_cache function set); architectures register themselves in the ModelRegistry. Phase 1 ships the Llama family (incl. SmolLM2).

Structs

ArchSpec

Resolved architecture description consumed by the decoder.

CacheConfig

Configuration for a KV cache instance.

ContiguousKVCache

Simple contiguous cache: stores one K and one V tensor per layer and concatenates along the sequence dimension every step.

LlamaModel

Llama-family causal LM, parameterized by the resolved ArchSpec — llama, smollm2, qwen2, and mistral today; the gemma/qwen3/phi presets migrate onto it stage by stage (roadmap wave 2).

ModelRegistry

Maps architecture identifiers (config.json::model_type, plus known aliases) to loaders. Mirrors MLC’s model.py::MODELS table.

PageStats

A paged cache’s page-table state at a point in time (for monitoring).

PagedKVCache

Q4KWeight

A weight matrix resident in VRAM in packed Q4_K form ([n_out, k], k % 256 == 0, superblocks along k).

Q5KWeight

A weight matrix resident in VRAM in packed Q6_K form ([n_out, k], k % 256 == 0, superblocks along k). A weight matrix resident in VRAM in packed Q5_K form ([n_out, k], k % 256 == 0, superblocks along k).

Q6KWeight

Q40Weight

A weight matrix resident in VRAM in packed Q4_0 form. [n_out, k] row-major, k % 32 == 0, blocks along k — exactly the GGUF tensor layout, so from_gguf_bytes takes the mmap’d tensor bytes unchanged.

Q50Weight

A weight matrix resident in VRAM in packed Q5_0 form ([n_out, k], k % 32 == 0, blocks along k).

Q80Weight

A weight matrix resident in VRAM in packed Q8_0 form ([n_out, k], k % 32 == 0, blocks along k).

QuantizedLinear

y = x @ W^T (+ b) where W is stored group-quantized to 4 bits.

RotaryEmbedding

Precomputed RoPE cosine/sine tables.

SmolVlmModel

SmolVLM (Idefics3): SigLIP + connector + Llama-family text decoder.

WhisperModel

Enums

CacheKind

Which KVCache implementation to instantiate.

LayerKind

Attention kind of one layer.

Linear

A linear layer weight: dense tensor (portable path) or packed quant blocks bound to a fused device kernel.

ModelError

Errors produced while constructing or running models.

NormFlavor

RMSNorm flavor: plain (x̂·w) or gemma’s zero-centered (x̂·(1+w)).

Traits

GenerativeModel

Fixed contract every generative architecture implements — the direct analog of MLC’s embed / prefill / decode / create_kv_cache function set. The runtime only ever talks to models through this trait.

KVCache

Per-layer key/value storage that owns the attention computation.

QuantLinearOp

A backend-specific quantized-linear forward. Boxed into Linear::Quant at load time by try_quant_linear.

SpeechToTextModel

Speech-to-text models (Whisper-style encoder–decoder). A separate contract from GenerativeModel: the encoder runs once per audio window, then the decoder is stepped over token prefixes against the fixed encoder states.

Functions

dequantize_q4_0_gpu

Runs the dequant-only kernel over a raw Q4_0 block stream. Exists for validation (bit-exact vs the CPU reference) and debugging, not the hot path.

dequantize_q4_k_gpu

Runs the Q4_K dequant-only kernel (validation/debugging path).

dequantize_q5_0_gpu

Runs the Q5_0 dequant-only kernel (validation/debugging path).

dequantize_q5_k_gpu

Runs the Q5_K dequant-only kernel (validation/debugging path).

dequantize_q6_k_gpu

Runs the Q6_K dequant-only kernel (validation/debugging path).

dequantize_q8_0_gpu

Runs the Q8_0 dequant-only kernel (validation/debugging path).

image_prompt_expansion

Builds the Idefics3 prompt expansion for one image: <fake_token_around_image><global-img><image>×image_seq_len<fake_token_around_image>. (image_seq_len = 64 for SmolVLM-256M.) The returned string is meant to replace each <image> placeholder in the chat text.

load_speech_model

Loads a speech model by metadata.architecture — the ASR counterpart of the text registry’s loader map.

pixels_to_tensor

Builds a [1, 3, H, W] pixel tensor from planar CHW f32 data (used by the runtime to hand media to embed_multimodal).

repack_q4_0

Layout step: repack a raw GGUF Q4_0 block stream into the device layout the kernels consume — nibble bytes as little-endian u32 words (4 words per block) and one f32 scale per block. The f16→f32 scale conversion is exact, so no precision is lost relative to the reference.

repack_q4_k

Layout step for Q4_K: split each 144-byte superblock into SoA device arrays — (qs words, [d, dmin] f32 pairs, scale words). 148 B per 256 weights = 4.63 bits/weight (GGUF native is 4.5).

repack_q5_0

Layout step for Q5_0: SoA of (nibble words [4/blk], high-bit words [1/blk], f32 scales) — 24 B / 32 weights = 6.0 bits/weight.

repack_q6_k

Layout step for Q6_K: split each 210-byte superblock into SoA device arrays — (ql words, qh words, i8 scale words, d f32). 212 B per 256 weights = 6.63 bits/weight (GGUF native is 6.56).

repack_q8_0

Layout step for Q8_0: SoA of (i8 words [8/blk], f32 scales) — 36 B / 32 weights = 9.0 bits/weight.

rms_norm

y = x / rms(x) * w where rms is taken over the last dimension and eps is added inside the square root.

try_quant_linear

Type Aliases

Result

Convenient result alias for this crate.