Module trainer 

Core training loop responsible for producing tokenizer.json artefacts.

Structs

SequenceStream

Iterator wrapper that optionally tracks the total number of available sequences.

Trainer

High-level façade configuring and executing BPE training runs.

TrainerArtifacts

Artifacts returned after a training session completes.