Expand description
Native DTG-MA skill bake (Patent 2) — no Python, no torch.
The certified recipe of converter/make_skill_l1fcd.py, in Rust on
the FcdModel f32 replica:
- Phase A — a trainable L1 mask over FFN neurons (one logit per neuron, applied to the input of down_proj as σ(m)): pure LM loss on the task corpus + a progressive L1 penalty. Every 30 steps the binarized mask (σ>τ) is scored on held-out chunks; the best checkpoint — the denoising bottom — is restored at the end. Pruning noise neurons IMPROVES the model before it starts to hurt.
- Phase B — FCD: the FFN of the last N layers trains against the same LM loss with the hard mask active (cosine LR), held-out gated, best checkpoint restored.
Attention (softmax and GDN alike) is FROZEN and carries no gradient
— exactly like the reference recipe (torch.no_grad() around the
attention branch): the backward walks the residual stream through
the FFN chain only, which is what makes a pure-Rust backward small.
Structs§
- Bake
Artifacts - The trained artifacts: everything the defrag writer needs, f32.
- Bake
Hyper - Hyper-parameters — defaults are the certified recipe.
- Bake
Report - What the bake measured and produced.
Functions§
- skill_
bake - The whole recipe.
logreceives progress lines.