gproxy-tokenize-2.1.6
- gproxy-tokenize 2.1.6
- Docs.rs crate page
- MIT
- 27 July 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- LeenHawk
- Dependencies
- anyhow ^1 normal optional
  async-trait ^0.1 normal optional
  bytes ^1 normal optional
  dashmap ^6 normal optional
  http ^1 normal optional
  serde_json ^1 normal
  tiktoken-rs ^0.12 normal optional
  tokenizers ^0.23 normal optional
  tokio ^1 normal optional
  tracing ^0.1 normal optional
  tokio ^1 dev
- Versions
- 45.45% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

All

gproxy_tokenize2.1.6

Crate Items

Structs
Enums
Traits
Functions
Type Aliases

List of all items

Structs

tokenize::TokenizerRegistry
tokenize::VocabInfo

Enums

tokenize::VocabSource

Traits

tokenize::TokenizerClient
tokenize::TokenizerStore

Functions

tokenize::count
tokenize::count_text
tokenize::harvest
tokenize::is_gpt_family

Type Aliases

tokenize::RegistryHandle