Module tokenize 

Local token counting (§6.3): tiktoken for gpt families, bundled/downloaded HF tokenizers for the rest, char-estimate floor. Native-only behind count-local except the estimate, which serves the edge build.

Structs

CountResult

TokenizerRegistry

Global tokenizer registry living on AppState.

VocabInfo

Listing entry for the admin surface.

Enums

CountError

CountMethod

CountWarning

LoadRequestStatus

VocabSource

Where a vocab comes from.

Traits

TokenizerClient

TokenizerStore

Functions

count

Count tokens of a provider-native request body. map = provider settings tokenizer_map (glob → vocab name). Never fails: worst case is the chars/2 estimate.

count_detailed

Count with provenance and diagnostics. Unlike count, malformed JSON is never indistinguishable from a genuinely empty request: it falls back to a conservative estimate over the raw body and records warnings.

count_text

Count a single text buffer with the same local fallback Clove uses for Claude Web usage synthesis: cl100k when local tokenizers are enabled, otherwise the cross-target character estimate.

harvest

Harvest human-text from any provider-native request JSON: walks the value, collecting strings under text-ish keys (text, content, instructions, string-form system, gemini parts text) plus tool definitions serialized. Returns (texts, message_count) where message_count is the length of the largest messages / contents / input array found (0 if none).

is_gpt_family

Whether model belongs to a gpt family with an exact local tiktoken vocabulary (drives the §17 counting-ladder source label).

try_count

Strict counting for correctness-sensitive paths. This rejects malformed JSON and, with hf-registry, waits for store hydration/download instead of returning a result that changes after a background load.

try_harvest

Fallible harvesting for callers that must distinguish malformed JSON from an intentionally empty request.

Type Aliases

RegistryHandle

What count receives as the registry: a real handle under count-local, a unit on builds without it (edge) so call sites stay uniform.