ffbpe-0.1.9
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

ffbpe 0.1.9

Unicode-aware, streaming BPE training and tiktoken-compatible encoding

Crate
Source
Builds
Feature flags

..
encoder.md
index.md
model.md
pretokenizer.md
tiktoken.md
trainer.md
training.md

1
2
3

# tiktoken compatibility

::: ffbpe.tiktoken_compat.Encoding