riptoken 0.1.0

Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# Rust
target/
Cargo.lock.bak

# Python
__pycache__/
*.py[cod]
*$py.class
*.so
*.egg-info/
dist/
build/
.venv/
venv/
.pytest_cache/
.mypy_cache/
.ruff_cache/

# Maturin
wheels/

# Editors
.vscode/
.idea/
*.swp
*.swo
.DS_Store

# Vocab files (large, download separately)
*.tiktoken
!tests/data/*.tiktoken