dynamo-tokenizers-1.5.3
- dynamo-tokenizers 1.5.3
- Permalink
- Docs.rs crate page
- Apache-2.0
- 14 July 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- nv-anants
- saturley-hall
- pvijayakrish
- nv-nmailhot
- Dependencies
- aho-corasick ^1 normal
  anyhow ^1 normal
  base64 ^0.22 normal
  blake3 ^1 normal
  fastokens ^0.2.0 normal
  moka ^0.12 normal
  rayon ^1 normal
  rustc-hash ^1.1 normal
  serde ^1 normal
  serde_json ^1 normal
  strum ^0.27 normal
  tiktoken-rs ^0.9 normal
  tokenizers ^0.21.4 normal
  tracing ^0.1 normal
  tempfile ^3 dev
- Versions
- 54.63% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

Offsets

dynamo_tokenizers1.5.3

dynamo_tokenizers

Type Alias Offsets

pub type Offsets = (usize, usize);

Expand description

character offsets in the original text