gpt_tokenizer-0.1.0
- gpt_tokenizer 0.1.0
- Permalink
- Docs.rs crate page
- MIT
- 05 August 2026
- Links
- Repository
- crates.io
- Source
- Owners
- cloudbridgeuy
- Dependencies
- regex ^1.7.1 normal
  serde_json ^1.0.93 normal
- Versions
- 64.29% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

All

gpt_tokenizer0.1.0

Crate Items

Structs
Constants
Functions

List of all items

Structs

Default

Functions

bpe_ranks
bytes_to_unicode
decode
encode

Constants

ENCODER_JSON
VOCAB_BPE