hermes-train

Burn-native training for the same MAL-driven Transformer used by hermes-llm inference. There is no Python model mirror or checkpoint adapter.

Build

# CPU
cargo build --release -p hermes-train

# Apple Metal
cargo build --release -p hermes-train --features metal

# NVIDIA CUDA
cargo build --release -p hermes-train --features cuda

Train

hermes-train train \
  --config models/hybrid-tiny.mal \
  --tokenizer tokenizer.json \
  --data corpus.jsonl \
  --output checkpoint \
  --batch-size 8 \
  --grad-accum 4 \
  --shuffle-buffer 8192 \
  --checkpoint-every 100 \
  --seq-len 256 \
  --epochs 1

Training data is either a text file or JSONL with a string text field; both formats may be Zstandard-compressed (.zst). The reader streams documents and fixed-length samples through a deterministic bounded shuffle buffer instead of retaining the corpus in memory. Repeat --data to combine files. Samples never cross document boundaries. Set --shuffle-buffer 0 only for ordered diagnostic runs.

The trainer uses Burn Autodiff with Burn's Muon optimizer for hidden 2D matrices and AdamW for embeddings, output weights, norms, biases, and convolution kernels. This matches the original trainer's optimizer split, including Muon's 20x learning rate, AdamW beta2 of 0.95, and global norm clipping. It supports cosine or warmup-stable-decay scheduling and fine-tuning from a Burn-native checkpoint. It atomically replaces the latest native checkpoint every 100 optimizer steps by default; pass --checkpoint-every 0 to save only at completion. On Mamba models, training and inference use fused CubeCL selective-scan kernels on Metal and CUDA; CPU uses the tensor-operation reference implementation.

Outputs are deliberately minimal:

• config.json, with tokenizer vocabulary size applied
• metrics.jsonl, flushed after every optimizer step for live reporters
• weights.safetensors, using Burn-native parameter names

The checkpoint loads directly in hermes-llm with strict tensor and shape validation. Experiment services such as W&B can tail metrics.jsonl without being linked into the training process.