gproxy-tokenize 2.6.4

Offline token counting for LLM request bodies: tiktoken, Hugging Face vocabularies, character estimate
Documentation
1
2
3
4
5
6
7
8
//! GPROXY local token counting.
//!
//! The nested `tokenize` module preserves the pre-split module paths. The root
//! re-exports keep the crate ergonomic for direct users.

pub mod tokenize;

pub use tokenize::*;