gproxy-tokenize

Count tokens of an LLM request body locally, without calling the provider.

gproxy-tokenize is the local token-counting layer of GPROXY. It takes a provider-native JSON body (OpenAI chat/responses, Claude messages, Gemini generateContent), harvests the text it contains, and counts it.

Usage

[dependencies]
gproxy-tokenize = { version = "2", features = ["count-local"] }

use gproxy_tokenize::{count, count_text};

// A single text buffer.
let n = count_text("Hello, how are you today?");

// A provider-native request body, counted against a model.
let body = br#"{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}"#;
let n = count("gpt-4o-mini", body, None, &registry);

registry is a &TokenizerRegistry (see below). Without the count-local feature it is () instead, so call sites stay identical across native and edge builds.

count never fails. Worst case it degrades to the character estimate, so a counting path can always produce a number.

Counting ladder

tiktoken for the gpt families (gpt-3.5, gpt-4, gpt-4o, gpt-4.1, gpt-5, o1, o3, o4) — exact vocabularies compiled in.
Hugging Face tokenizers for everything else, resolved through a TokenizerRegistry. A tokenizer_map of glob → vocab name can redirect a model to a specific vocabulary; otherwise the model name itself is looked up.
Bundled fallback vocabulary when the model resolves to nothing.
Character estimate (chars / 2) as the floor.

Each message adds a fixed framing overhead, so the result approximates what a provider bills rather than raw text tokens.

Registry

TokenizerRegistry is the cache for Hugging Face vocabularies. It is wired to the host application through two traits, so this crate itself performs no I/O:

TokenizerStore — persistence (list / get / put a vocabulary by name).
TokenizerClient — outbound HTTP, used to hydrate a missing vocabulary.

Lookups are non-blocking: a miss returns None and schedules a background load via request_load, while the caller falls further down the ladder.

Features

Feature	Default	Effect
`count-local`	off	tiktoken + Hugging Face `tokenizers` + the registry. Native targets.

With count-local off, the crate has a single dependency (serde_json) and compiles to wasm32; only harvest, is_gpt_family, and the character estimate remain. This is the edge/serverless build.

License

Licensed under the MIT License.