Skip to main content

apply_chat_profile_env

Function apply_chat_profile_env 

Source
pub fn apply_chat_profile_env(snapshot_path: &Path)
Expand description

Chat-profile runtime defaults for ferrum run. Sniffs the arch (dense vs MoE — works for both GGUF files and safetensors snapshot dirs) and materializes missing compatibility env vars for:

  • FERRUM_KV_CAPACITY — 8192 dense / 4096 MoE
  • FERRUM_PAGED_KV / legacy FERRUM_METAL_PAGED_KV — 0 GGUF / 1 only for Qwen3 dense and MoE safetensors. The Metal Qwen3-MoE GGUF paged-KV decode path can repeat the first generated token until max_tokens; keep GGUF on the contiguous path until that kernel path is fixed. Qwen3 dense safetensors is validated on paged KV; TinyLlama/Llama and Qwen2 dense safetensors produce token noise on the Metal paged-KV path and default to contiguous KV.
  • FERRUM_PAGED_MAX_SEQS=2 dense / 1 MoE, FERRUM_MAX_BATCH=1 — single-user REPL. Keeps the paged pool at ~1.7 GB for cap=8192 dense; without this cap the default max_seqs=32 makes the pool ~30 GB on a 32 GB Mac.
  • FERRUM_MOE_BATCHED=0, FERRUM_MOE_BATCHED_DECODE=0, FERRUM_MOE_BATCH_THRESHOLD=2 — MoE only. run is an interactive single-session path, so do not engage unneeded multi-sequence MoE batching.

Idempotent: if a user explicitly sets one of these env vars before invoking ferrum run, that value wins (we only set when unset). Called automatically by resolve_model_source when the autosize profile is Chat. Server/bench callers don’t get these defaults — they don’t fit the multi-turn REPL pattern this profile is tuned for.

Without this, dense safetensors models (e.g. Qwen/Qwen3-0.6B) inherit the model-level DEFAULT_KV_CAPACITY=512 floor in llama_family.rs::ensure_kv, which overflows after ~512 tokens on a max_tokens=2048 chat — manifesting as a KV cache overflow on layer 0 panic mid-response.