Module tokenization 

Shared tokenization utilities for indexing and search.

Used by the ContextIndex and search paths to ensure consistent token generation across all search operations.

Constants

MAX_INDEX_TOKENS

MAX_JSON_TOKEN_BUDGET

MAX_TOKEN_LEN

Functions

push_json_tokens

push_text_tokens

push_value_tokens