pub fn apply_chat_profile_env(snapshot_path: &Path)Expand description
Chat-profile runtime defaults for ferrum run. Sniffs the arch
(dense vs MoE — works for both GGUF files and safetensors snapshot
dirs) and materializes missing compatibility env vars for:
FERRUM_KV_CAPACITY— 8192 dense / 4096 MoEFERRUM_PAGED_KV/ legacyFERRUM_METAL_PAGED_KV— 0 GGUF / 1 only for Qwen3 dense and MoE safetensors. The Metal Qwen3-MoE GGUF paged-KV decode path can repeat the first generated token untilmax_tokens; keep GGUF on the contiguous path until that kernel path is fixed. Qwen3 dense safetensors is validated on paged KV; TinyLlama/Llama and Qwen2 dense safetensors produce token noise on the Metal paged-KV path and default to contiguous KV.FERRUM_PAGED_MAX_SEQS=2dense /1MoE,FERRUM_MAX_BATCH=1— single-user REPL. Keeps the paged pool at ~1.7 GB forcap=8192dense; without this cap the defaultmax_seqs=32makes the pool ~30 GB on a 32 GB Mac.FERRUM_MOE_BATCHED=0,FERRUM_MOE_BATCHED_DECODE=0,FERRUM_MOE_BATCH_THRESHOLD=2— MoE only.runis an interactive single-session path, so do not engage unneeded multi-sequence MoE batching.
Idempotent: if a user explicitly sets one of these env vars before
invoking ferrum run, that value wins (we only set when unset).
Called automatically by resolve_model_source when the autosize
profile is Chat. Server/bench callers don’t get these defaults —
they don’t fit the multi-turn REPL pattern this profile is tuned for.
Without this, dense safetensors models (e.g. Qwen/Qwen3-0.6B)
inherit the model-level DEFAULT_KV_CAPACITY=512 floor in
llama_family.rs::ensure_kv, which overflows after ~512 tokens on
a max_tokens=2048 chat — manifesting as a KV cache overflow on layer 0 panic mid-response.