dynamo-tokenizers-1.2.1
- dynamo-tokenizers 1.2.1
- Docs.rs crate page
- Apache-2.0
- 13 June 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- nv-anants
- saturley-hall
- pvijayakrish
- nv-nmailhot
- Dependencies
- anyhow ^1 normal
  base64 ^0.22 normal
  fastokens ^0.1.0 normal
  rayon ^1 normal
  rustc-hash ^1.1 normal
  serde ^1 normal
  serde_json ^1 normal
  strum ^0.27 normal
  tiktoken-rs ^0.9 normal
  tokenizers ^0.21.4 normal
  tracing ^0.1 normal
  tempfile ^3 dev
- Versions
- 44.87% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

TokenIdType

dynamo_tokenizers1.2.1

dynamo_tokenizers

Type Alias TokenIdType

pub type TokenIdType = u32;