dynamo-tokenizers-1.5.3
- dynamo-tokenizers 1.5.3
- Permalink
- Docs.rs crate page
- Apache-2.0
- 14 July 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- nv-anants
- saturley-hall
- pvijayakrish
- nv-nmailhot
- Dependencies
- aho-corasick ^1 normal
  anyhow ^1 normal
  base64 ^0.22 normal
  blake3 ^1 normal
  fastokens ^0.2.0 normal
  moka ^0.12 normal
  rayon ^1 normal
  rustc-hash ^1.1 normal
  serde ^1 normal
  serde_json ^1 normal
  strum ^0.27 normal
  tiktoken-rs ^0.9 normal
  tokenizers ^0.21.4 normal
  tracing ^0.1 normal
  tempfile ^3 dev
- Versions
- 54.63% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

Module tiktoken

dynamo_tokenizers1.5.3

Module tiktoken

Module Items

Structs

In crate dynamo_tokenizers

dynamo_tokenizers

Module tiktoken

Structs§

TikTokenTokenizer