Skip to main content

Module fcd

Module fcd 

Source
Expand description

FCD polish trainer — native Rust quality-polish for O(1)-converted models (docs/RUST_FCD.md). Removes the last Python dependency from the cortiq convert --o1 pipeline.

Certified recipe (torch reference nystrom_fcd_full2_06b.py, Qwen3-0.6B 28/28): train ONLY the LN gains + FFN of the converted layers, loss = (1−0.7)·CE + 0.7·KL(teacher‖student), AdamW 5e-5 (torch defaults), grad clip 1.0, batch 2×512 fresh random windows, quick deterministic val every 25 steps, restore the best checkpoint.

Structure:

  • the whole model is dequantized to f32 once; teacher and student SHARE the frozen set, trainables get separate master copies (the KL anchor never drifts);
  • layer-level activation checkpointing: the forward keeps only each layer’s input hidden; the backward re-runs one layer at a time;
  • converted layers use the certified matrix form of the Nyström joint kernel in f64 (fcd_ops), M constant in backward;
  • GDN (GatedDeltaNet) layers of Qwen3.5-class hybrids run frozen in BOTH teacher and student, with a true BPTT through-backward (fcd_ops::gdn_*) so trainable layers BELOW them still learn; vmf_phase linear layers are refused (no backward yet).

Structs§

FcdHyper
Training hyper-parameters (defaults = the certified recipe).
FcdModel
The f32 training replica of a .cmf model (≤ 1B targets).
FcdReport
What the polish measured — written into provenance.fcd and reported by the CLI.
GateReport
What the generation gate saw and decided.
GenGateCfg
Generation-gate configuration (Patent 16 draft, claim 13: checkpoint selection gated on generation-behavior metrics measured through the SERVED kernel, not on the training objective alone).
TrainState
Master copies + grads + AdamW moments of the trainable tensors.

Functions§

gate_pass
Gate predicate (Patent 16 draft, claim 13): a checkpoint PASSES iff no prompt’s loop score exceeds threshold AND none exceeds its zero-shot baseline by more than slack — boundary values pass.
loop_score
Loopiness of a generated id sequence: 1 − unique 4-grams / total. 0 = no repeated 4-gram; near 1 = a tight loop.
run_polish
Run the full certified polish: train, early-stop/restore-best, and write <out> (source tensors byte-copied, polished LN/FFN as f32).
select_checkpoint
Checkpoint selection: lowest val ppl AMONG GATE-PASSING checkpoints (ties → earliest). None = nothing passed → the caller must restore the zero-shot state (identity polish): the stage must never make generation worse than conversion alone. (Patent 16 draft, claim 13.)