unitoken 0.1.5

Fast BPE tokenizer/trainer with a Rust core and Python bindings
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/target
Cargo.lock
out/
dist/
*.svg
.cache
uv.lock

_lib.*.so
_lib.*.pyd
__pycache__/
merges.txt
vocab.json
merges.*.txt
vocab.*.json