Skip to main content

Crate gproxy_tokenize

Crate gproxy_tokenize 

Source
Expand description

GPROXY local token counting.

The nested tokenize module preserves the pre-split module paths. The root re-exports keep the crate ergonomic for direct users.

Re-exports§

pub use tokenize::*;

Modules§

tokenize
Local token counting (§6.3): tiktoken for gpt families, bundled/downloaded HF tokenizers for the rest, char-estimate floor. Native-only behind count-local except the estimate, which serves the edge build.