Crate kopitiam_runtime 

Kopitiam Runtime: the Qwen-family transformer forward pass, running entirely in Rust on CPU, offline.

This crate is the payoff of the layers beneath it. kopitiam-core defines the shared vocabulary (kopitiam_core::DType, kopitiam_core::Shape); kopitiam-tensor implements the CPU tensor and its ops (matmul, softmax, RMSNorm, SiLU, embedding gather, GGUF block-quantization decoding); kopitiam-loader parses GGUF/SafeTensors files into raw bytes plus dtype/shape metadata; kopitiam-tokenizer turns text into token ids and back. None of those crates constructs a model or runs a forward pass — this one does.

Pipeline

kopitiam_loader::load_model(path)
  -> LoadedModel
  -> QwenModel::from_loaded_model         (crate::model)
       - QwenConfig::from_metadata        (crate::config)
       - ModelWeights::load               (crate::weights, via crate::bridge)
       - RotaryEmbedding::new             (crate::rope)
  -> generate(&model, &tokenizer, prompt, ...)   (crate::generate)
       per new token:
         embedding lookup                  (Tensor::gather_rows)
         per layer: block_forward           (crate::block)
           RMSNorm -> attention_forward     (crate::attention)
             RoPE (split-half)              (crate::rope)
             grouped-query KV repeat        (crate::attention::repeat_kv_heads)
             causal mask + softmax
             KV cache read/append           (crate::kv_cache)
           RMSNorm -> swiglu_mlp            (crate::mlp)
         final RMSNorm -> output projection
         [optional] constraint mask -> -inf  (crate::constraint)
         greedy sampling                    (crate::sampling)

Module map

• [bridge] — the loader/tensor glue: bytes + dtype + shape -> Tensor. kopitiam-loader and kopitiam-tensor were built concurrently and deliberately do not depend on each other; this crate is the first one downstream of both, so this is where that gap is bridged.
• [config] — config::QwenConfig, the architecture hyperparameters resolved (with documented fallbacks and validation) from kopitiam_loader::ModelMetadata.
• [rope] — rotary position embeddings, split-half (“NEOX”) pairing. Read this module’s docs before touching anything position-related; a swapped pairing convention is this crate’s single easiest place to introduce silent, undetectable-by-type-system wrongness.
• [kv_cache] — the per-layer, growable key/value cache that makes autoregressive decoding linear instead of quadratic in sequence length.
• [attention] — grouped-query causal self-attention: repeating shared KV heads across their query-head group, causal masking, and wiring RoPE and the KV cache into one attention forward pass.
• [mlp] — the SwiGLU feed-forward block.
• [linear] — the single x @ W^T + b helper every projection in this crate goes through.
• [block] — one pre-norm transformer block (attention sub-layer, MLP sub-layer, both with a residual connection).
• [weights] — loads every named GGUF weight tensor a block/model needs.
• [model] — model::QwenModel: wires embedding, every block, the final norm, and the (possibly tied) output projection into a traits::Model implementation.
• traits — traits::Model and traits::Backend, the Model Runtime boundary every layer above this crate should code against.
• [sampling] — turning a row of logits into a token id: sampling::GreedySampler (argmax) and sampling::StochasticSampler (temperature/top-k/top-p/min-p/ repetition penalty, composed as a pipeline and driven by a seeded PRNG — see that module’s docs for the pipeline shape and why seedability is mandatory, not optional).
• [constraint] — grammar-constrained decoding (the keystone): mask the tokens a constraint::TokenConstraint forbids to -inf at the front of the sampling path, before temperature/top-k/top-p, so the model physically cannot emit invalid structure. Ships a fixed allowed-token-set (constraint::AllowedTokens, e.g. a tool-name enum) and a structural JSON constraint (constraint::JsonStructure); constraint::mask_logits is the masking step and constraint::ConstrainedSampler the drop-in sampler wrapper, driven end to end by generate::generate_constrained. See that module’s docs for why mask-before-sample and -inf-not-0.0 (AID-0045).
• [gguf_tokenizer] — builds a kopitiam_tokenizer::BpeTokenizer directly from a GGUF file’s embedded tokenizer.ggml.* vocabulary (no companion tokenizer.json needed).
• generate — the end-to-end entry point: prompt -> tokens -> forward passes -> sampled ids -> text, with streaming per-token output.

What is here as of Phase 2, and what is still deliberately not

As of Phase 2: stochastic sampling (sampling::StochasticSampler — temperature/top-k/top-p/min-p/repetition penalty) alongside greedy, and a fused quantized matmul for Q4_0/Q8_0 matmul-operand weights (see kopitiam_tensor::Tensor::quantized_matmul and bridge::load_matmul_weight — weights whose on-disk dtype is quantized now stay quantized in memory instead of being eagerly dequantized to f32, which is what makes a multi-gigabyte model’s resident memory footprint match its on-disk size rather than balloon by 4-8x).

Still not here: no batching across multiple concurrent prompts; no scheduler or execution graph (see the parent epic, kopitiam-082, Phase 3, and this crate’s benchmark module for why a general graph engine was judged not to earn its keep yet); no SIMD. “Correct before fast” per this workspace’s engineering principles remains the ordering principle — every fast path added so far (quantized matmul) ships alongside, and is tested against, the plain reference it replaces.

Re-exports

pub use traits::Backend;

pub use traits::CpuBackend;

pub use traits::Model;

Modules

traits

The Model Runtime boundary: Model and Backend.

Structs

AllowedTokens

A fixed allowed-token-set constraint: the same ids every step, regardless of decode state.

ConstrainedSampler

Wraps any Sampler so the constraint’s mask is applied at the front of the sampling path — the drop-in integration seam for real decoding.

DecodeState

The decode state a TokenConstraint reasons about when it decides which next tokens are valid.

GenerationConfig

Generation limits and stop conditions for generate.

GreedySampler

Always picks the highest-scoring token: argmax(logits).

JsonStructure

A TokenConstraint that keeps the model’s output structurally valid JSON.

KvCache

Per-layer, growable key/value cache for one generation session.

QwenConfig

The resolved shape of a Qwen-family transformer: everything the forward pass needs to know about the model it is about to run, with no further Options or metadata lookups once construction succeeds.

QwenModel

A loaded, ready-to-run Qwen-family transformer.

RotaryEmbedding

Precomputed cos/sin tables for every position up to max_position, so RotaryEmbedding::apply is a table lookup per (position, pair) rather than a cos/sin call — cheap enough to matter once this runs once per token per layer per head during decode.

SamplingConfig

Configuration for StochasticSampler’s logit-transform pipeline.

Shape

The dimensions of a tensor, outermost first.

SliceVocab

The obvious TokenVocab over an owned id -> bytes table. Handy for tests and for any caller that already has the vocabulary as a Vec.

StochasticSampler

Temperature / top-k / top-p / min-p / repetition-penalty sampling, composed as a pipeline of logit transforms (see this module’s docs) and driven by a seeded [Rng].

Enums

AllowedSet

Which next-token ids a constraint permits at the current step.

ConstrainedGenerateError

Either a normal runtime error, or the constraint leaving no valid token — the two failure modes generate_constrained can hit.

ConstraintError

What went wrong when a constraint could not be satisfied.

DType

The element type of a tensor.

Device

Where a tensor’s storage lives and where kernels run on it.

Error

Every way a Kopitiam Runtime operation can fail.

Traits

Sampler

Chooses the next token id from one row of logits (length vocab_size).

TokenConstraint

A rule that says which next tokens are valid, given what’s been decoded so far.

TokenVocab

A token id -> its byte string, for constraints that must reason about the bytes a candidate token would append (not just its id).

Functions

generate

Greedily generates a completion for prompt.

generate_constrained

Like generate_with_sampler, but every step is grammar-constrained: the ConstrainedSampler’s TokenConstraint masks disallowed tokens to -inf before its inner sampler runs, so the model physically cannot emit a token the constraint forbids. This is the keystone that makes a small model reliably produce valid JSON / valid tool names / valid structure — see [crate::constraint] for the full rationale and provenance (AID-0045).

generate_with_sampler

Identical to generate except the token-selection strategy is pluggable: any Sampler impl (greedy, or a crate::sampling::StochasticSampler configured for temperature/ top-k/top-p/min-p/repetition-penalty sampling) drives which token is picked at every step, instead of always argmax. Every other detail — prefill, one-token-at-a-time KV-cache decoding, EOS handling, streaming via on_token, the returned completion text — is exactly generate’s behaviour, because generate is defined in terms of this function; see this module’s docs for why that direction of composition (not the reverse) is what keeps the two entry points from drifting apart.

greedy_argmax

argmax(logits), ties broken toward the lowest id (the first maximum encountered) — a PartialOrd tie-break rule that is total and deterministic even in the presence of NaN (f32::partial_cmp’s None case), unlike an assert-free .max_by(...) over raw f32, which would panic or silently misbehave on a NaN logit.

load_matmul_weight

Loads name the way every matmul-operand weight (wq/wk/wv/wo, the SwiGLU MLP’s three matrices, and output.weight — see [crate::weights::LayerWeights] / [crate::weights::ModelWeights]) should: keeping a block-quantized on-disk dtype only when there is a fused kernel to compute on it, and otherwise dequantizing it to f32 at load.

load_matmul_weight_opt

Like load_matmul_weight, but returns Ok(None) instead of Error::MissingTensor when name is absent — the matmul-operand counterpart to load_tensor_f32_opt, for the same optional weights (per-projection attention biases, output.weight when tied to the token embedding).

load_tensor_f32

Looks up name in model, builds a Tensor from its bytes, and dequantizes/upcasts it to f32 — the dtype every op in kopitiam-tensor actually computes in. This is the call every weight-loading site in [crate::weights] goes through: model files may ship f16, bf16, or any of the five GGUF block-quantized formats, and this one function makes the rest of the forward pass indifferent to which.

load_tensor_f32_opt

Like load_tensor_f32, but returns Ok(None) instead of Error::MissingTensor when name is absent.

mask_logits

The masking step itself: set every logit whose id is not in allowed to f32::NEG_INFINITY, in place.

tensor_from_entry

Builds a Tensor from one TensorEntry’s bytes, preserving its on-disk DType exactly (still block-quantized if entry.dtype is quantized, still f16/bf16 if the file stored it that way).

tokenizer_from_gguf

Builds a BpeTokenizer from model’s embedded GGUF vocabulary.

Type Aliases

Result

The runtime’s result alias.