ffbpe 0.1.10

Unicode-aware, streaming BPE training and tiktoken-compatible encoding
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# Python API

The public Python package is organized around one shortcut and four core
objects:

| API | Purpose |
|---|---|
| [`train_bpe()`][ffbpe.training.train_bpe] | Train directly from a string or iterable |
| [`PreTokenizer`][ffbpe.pretokenizer.PreTokenizer] | Count words and Unicode bigrams |
| [`BpeTrainer`][ffbpe.trainer.BpeTrainer] | Train from an explicit word inventory |
| [`BpeModel`][ffbpe.model.BpeModel] | Validate, encode, and save a trained model |
| [`BpeEncoder`][ffbpe.encoder.BpeEncoder] | Load and use model files |

The generated reference follows the typed Python wrappers. Rust extension
implementation classes are intentionally omitted from the public docs.