riptoken-0.1.0
Platform
Feature flags

docs.rs

Rust

riptoken 0.1.0

Fast BPE tokenizer for LLMs — a faster, drop-in compatible reimplementation of tiktoken

Crate
Source
Builds
Feature flags

.cargo
.github
benches
python
src
tests
.cargo_vcs_info.json
.gitignore
Cargo.lock
Cargo.toml
Cargo.toml.orig
CHANGELOG.md
LICENSE
pyproject.toml
README.md
rust-toolchain.toml

1
2
3

[toolchain]
channel = "stable"
components = ["rustfmt", "clippy"]