dynamo-tokenizers-1.3.0
- dynamo-tokenizers 1.3.0
- Docs.rs crate page
- Apache-2.0
- 03 June 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- nv-anants
- saturley-hall
- pvijayakrish
- nv-nmailhot
- Dependencies
- aho-corasick ^1 normal
  anyhow ^1 normal
  base64 ^0.22 normal
  blake3 ^1 normal
  fastokens ^0.1.0 normal
  moka ^0.12 normal
  rayon ^1 normal
  rustc-hash ^1.1 normal
  serde ^1 normal
  serde_json ^1 normal
  strum ^0.27 normal
  tiktoken-rs ^0.9 normal
  tokenizers ^0.21.4 normal
  tracing ^0.1 normal
  tempfile ^3 dev
- Versions
- 50% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

All

dynamo_tokenizers1.3.0

Crate Items

Structs
Enums
Traits
Functions
Type Aliases

List of all items

Structs

DecodeStream
Error
Sequence
StopSequenceDecoder
StopSequenceDecoderBuilder
Tokenizer
cache::CachedTokenizer
cache::L1Cache
cache::L1CacheStats
fastokens::FastTokenizer
hf::HuggingFaceTokenizer
tiktoken::TikTokenTokenizer

Enums

Encoding
SequenceDecoderOutput
TokenizerType
traits::DecodeResult

Traits

traits::Decoder
traits::Encoder
traits::Tokenizer

Functions

create_tokenizer_from_file
file_json_field
hf::merge_special_tokens_from_config
log_json_err

Type Aliases

Offsets
Result
TokenIdType
cache::CacheEventFn