llm-tokenizer-1.3.2
- llm-tokenizer 1.3.2
- Docs.rs crate page
- Apache-2.0
- Links
- Repository
- crates.io
- Source
- Owners
- slin1237
- Dependencies
- aho-corasick ^1.1 normal
  anyhow ^1.0 normal
  base64 ^0.22 normal
  blake3 ^1.5 normal
  dashmap ^6.1.0 normal
  futures ^0.3 normal
  hf-hub ^0.5.0 normal
  minijinja ^2.0 normal
  minijinja-contrib ^2.0 normal
  rayon ^1.10 normal
  rustc-hash ^1.1 normal
  serde ^1.0 normal
  serde_json ^1.0 normal
  thiserror ^2.0.12 normal
  tiktoken-rs ^0.9.1 normal
  tokenizers ^0.22.0 normal
  tokio ^1.42.0 normal
  tracing ^0.1 normal
  uuid ^1.10 normal
  criterion ^0.8 dev
  openai-protocol ^1.7.0 dev
  reqwest ^0.12.8 dev
  tempfile ^3.8 dev
- Versions
- 73.5% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

TokenIdType

llm_tokenizer1.3.2

In llm_tokenizer::traits

llm_tokenizer::traits

Type Alias TokenIdType

pub type TokenIdType = u32;

Expand description

Type alias for token IDs