Module tokenize 

Local token counting (§6.3): tiktoken for gpt families, bundled/downloaded HF tokenizers for the rest, char-estimate floor. Native-only behind count-local except the estimate, which serves the edge build.

Structs

TokenizerRegistry

Global tokenizer registry living on AppState.

VocabInfo

Listing entry for the admin surface.

Enums

VocabSource

Where a vocab comes from.

Traits

TokenizerClient

TokenizerStore

Functions

count

Count tokens of a provider-native request body. map = provider settings tokenizer_map (glob → vocab name). Never fails: worst case is the chars/2 estimate.

count_text

Count a single text buffer with the same local fallback Clove uses for Claude Web usage synthesis: cl100k when local tokenizers are enabled, otherwise the cross-target character estimate.

harvest

Harvest human-text from any provider-native request JSON: walks the value, collecting strings under text-ish keys (text, content, instructions, string-form system, gemini parts text) plus tool definitions serialized. Returns (texts, message_count) where message_count is the length of the largest messages / contents / input array found (0 if none).

is_gpt_family

Whether model belongs to a gpt family with an exact local tiktoken vocabulary (drives the §17 counting-ladder source label).

Type Aliases

RegistryHandle

What count receives as the registry: a real handle under count-local, a unit on builds without it (edge) so call sites stay uniform.