Struct TransformerConfig 

pub struct TransformerConfig {

    pub hidden_size: usize,
    pub num_attention_heads: usize,
    pub num_kv_heads: usize,
    pub intermediate_size: usize,
    pub num_hidden_layers: usize,
    pub vocab_size: usize,
    pub max_position_embeddings: usize,
    pub rms_norm_eps: f32,
    pub rope_theta: f32,
    pub use_bias: bool,
    pub head_dim_override: Option<usize>,
    pub architecture: ModelArchitecture,
    pub hf_architecture: Option<String>,
    pub hf_model_type: Option<String>,
    pub tie_word_embeddings: bool,
}

Configuration for transformer models

Fields

hidden_size: usize

Hidden dimension (embedding size)

num_attention_heads: usize

Number of attention heads

num_kv_heads: usize

Number of key-value heads (for grouped-query attention)

intermediate_size: usize

Feed-forward network intermediate dimension

num_hidden_layers: usize

Number of transformer layers

vocab_size: usize

Vocabulary size

max_position_embeddings: usize

Maximum sequence length

rms_norm_eps: f32

RMS normalization epsilon

rope_theta: f32

RoPE theta base

use_bias: bool

Whether to use bias in linear layers

head_dim_override: Option<usize>

Explicit per-head dimension (overrides hidden_size / num_heads). Required for Qwen3 where head_dim=128 but hidden_size/num_heads=80.

architecture: ModelArchitecture

Architecture family: encoder (BERT/RoBERTa) or decoder (LLaMA/Qwen). Determines position encoding, normalization, activation, and pooling strategy.

hf_architecture: Option<String>

HuggingFace architecture class name (e.g., “Qwen2ForCausalLM”, “LlamaForCausalLM”). Used for checkpoint config.json compatibility.

hf_model_type: Option<String>

HuggingFace model type (e.g., “qwen2”, “llama”). Used for checkpoint config.json compatibility.

tie_word_embeddings: bool

Whether to tie input/output embeddings (embed_tokens and lm_head). Qwen2: true, LLaMA: false.

Implementations

impl TransformerConfig

pub fn llama2_7b() -> Self

LLaMA 2 7B configuration

pub fn llama2_13b() -> Self

LLaMA 2 13B configuration

pub fn mistral_7b() -> Self

Mistral 7B configuration

pub fn qwen2_0_5b() -> Self

Qwen2 0.5B configuration (good for testing).

Empirically verified against ~/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-0.5B-Instruct/.../config.json 2026-05-04. Pinned by contracts/apr-pretrain-arch-polymorphic-v1.yaml FALSIFY-001.

Note: tie_word_embeddings: true is the Qwen2.5 0.5B/1.5B convention (the 7B variant turns this OFF; see qwen2_7b()). This is a Qwen scaling-law quirk — small Qwen models reuse embedding+lm_head weights to save params, but the larger variants pay the param cost for untied weights. Drift-prevention: keeping this true is required for SHIP-TWO-001 §49 MODEL-2 fine-tune from a Qwen2.5-Coder-0.5B checkpoint.

pub fn qwen2_1_5b() -> Self

Qwen2.5-Coder-1.5B-Instruct: 28 layers, 12 heads, 2 KV heads, hidden=1536

pub fn qwen2_7b() -> Self

Qwen2.5-Coder 7B configuration (GH-371)

Qwen2.5-Coder-7B-Instruct: 28 layers, 28 heads, 4 KV heads, hidden=3584 Contract: contracts/model-families/qwen2.yaml

pub fn qwen3_4b() -> Self

Qwen3 4B configuration

Qwen3-4B: 36 layers, 32 heads, 8 KV heads, hidden=2560, head_dim=128. Same vocab_size as Qwen2 (151936). No attention bias (Qwen3 family).

pub fn qwen3_5_9b() -> Self

Qwen3.5 9B configuration

Key differences from Qwen2: no attention bias, head_dim=256 (explicit), vocab_size=248320, hybrid attention (standard + linear layers). Contract: contracts/model-families/qwen3_5.yaml

pub fn from_apr_metadata( hidden_size: Option<usize>, num_heads: Option<usize>, num_kv_heads: Option<usize>, intermediate_size: Option<usize>, num_layers: Option<usize>, vocab_size: Option<usize>, max_position_embeddings: Option<usize>, rms_norm_eps: Option<f32>, rope_theta: Option<f32>, architecture: Option<&str>, ) -> Option<Self>

Construct from APR v2 metadata fields.

CONTRACT: The .apr file is the single source of truth for model architecture. These fields were validated at import time by the tensor-layout-v1 contract. This function propagates that contract to the training pipeline — no hardcoded lookups, no silent fallbacks.

Returns None if any required field is missing, forcing the caller to handle the error explicitly rather than silently degrading to tiny().

GH-376: Fixes instruct pipeline ignoring .apr architecture metadata.

pub fn from_size_str(size: &str) -> Result<Self, String>

Resolve config from a model size string. Errors on unknown sizes.

GH-377: Replaces _ => TransformerConfig::tiny() catch-all pattern. This is the single canonical mapping from size strings to configs. Every callsite that previously had its own match table should use this.

pub fn codebert() -> Self

CodeBERT (microsoft/codebert-base) encoder configuration.

RoBERTa architecture: 12 layers, 768 hidden, 12 heads, GELU, LayerNorm, learned positions. SSC v11 Section 4: 125M params, ~20ms CPU inference, WASM-deployable.

pub fn tiny() -> Self

Tiny configuration for testing

pub fn is_encoder(&self) -> bool

Whether this config describes an encoder (BERT/RoBERTa) architecture.

pub fn hf_architecture_name(&self) -> &str

HuggingFace architecture class name for checkpoint config.json. Uses explicit override if set, otherwise infers from config.

pub fn hf_model_type_str(&self) -> &str

HuggingFace model_type string for checkpoint config.json.

pub fn ties_embeddings(&self) -> bool

Whether embeddings are tied (embed_tokens == lm_head). Uses explicit flag if set, otherwise infers from architecture.

pub fn head_dim(&self) -> usize

Per-head dimension.

Uses explicit override when set (Qwen3: head_dim=128 with hidden=2560, 32 heads). Falls back to hidden_size / num_heads for standard architectures.

pub fn q_dim(&self) -> usize

Total Q/O projection dimension = num_heads * head_dim.

Equals hidden_size for standard architectures but differs when head_dim is explicitly overridden (e.g. Qwen3-4B: 32 * 128 = 4096 != 2560).

pub fn per_layer_weight_elements(&self) -> usize

Per-layer weight VRAM in f32 elements (constant, independent of seq_len).

Maps to cuda_block.rs lines 212-220: GpuBuffer::from_host() uploads.

pub fn total_training_vram_bytes(&self, max_seq_len: usize) -> usize

Total VRAM in bytes for all layers at a given max_seq_len.

Postcondition: result is exact for the current cuda_block.rs buffer layout.

pub fn total_training_vram_bytes_shared(&self, max_seq_len: usize) -> usize

Total VRAM in bytes with SHARED scratch workspace (1 per model, not per layer).

This is the correct budget formula when gradient buffers are shared across layers (canonical in PyTorch/JAX). Only weights are truly per-layer.

Postcondition: result < total_training_vram_bytes(s) for L > 1

pub fn max_seq_len_for_vram_shared(&self, vram_bytes: usize) -> Option<usize>

Solve for the maximum seq_len that fits in the given VRAM budget (bytes), using shared scratch workspace.

This is the solver to use with the shared-scratch architecture. Returns None if even seq_len=1 exceeds the budget.

pub fn max_seq_len_for_vram(&self, vram_bytes: usize) -> Option<usize>

Solve for the maximum seq_len that fits in the given VRAM budget (bytes).

Binary search over [1, max_position_embeddings]. Returns None if even seq_len=1 exceeds the budget.

Precondition: vram_bytes > 0 Postcondition: total_training_vram_bytes(result) <= vram_bytes

Trait Implementations

impl Clone for TransformerConfig

fn clone(&self) -> TransformerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TransformerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TransformerConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TransformerConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.