hermes-train

Burn-native training for the same MAL-driven Transformer used by hermes-llm inference. There is no Python model mirror or checkpoint adapter.

Build

# CPU
cargo build --release -p hermes-train

# Apple Metal
cargo build --release -p hermes-train --features metal

# NVIDIA CUDA
cargo build --release -p hermes-train --features cuda

Train

hermes-train train \
  --config models/hybrid-tiny.mal \
  --tokenizer tokenizer.json \
  --data corpus.jsonl \
  --output checkpoints \
  --batch-size 8 \
  --grad-accum 4 \
  --checkpoint-every 100 \
  --seq-len 256 \
  --epochs 1

Training data is either a text file or JSONL with a string text field; both formats may be Zstandard-compressed (.zst). The reader streams documents and fixed-length samples instead of retaining the corpus in memory. Repeat --data to combine files. Samples never cross document boundaries.

The trainer uses Burn Autodiff and AdamW with norm clipping. It supports cosine or warmup-stable-decay scheduling and fine-tuning from a Burn-native checkpoint. It atomically replaces the latest native checkpoint every 100 optimizer steps by default; pass --checkpoint-every 0 to save only at completion. On Mamba models, training uses the differentiable tensor-operation selective scan; inference uses the custom CubeCL kernel on Metal/CUDA.

Outputs are deliberately minimal:

• config.json, with tokenizer vocabulary size applied
• metrics.jsonl, flushed after every optimizer step for live reporters
• weights.safetensors, using Burn-native parameter names

The checkpoint loads directly in hermes-llm with strict tensor and shape validation. Experiment services such as W&B can tail metrics.jsonl without being linked into the training process.