pub const CHARS_PER_TOKEN: usize = 3;Expand description
Conservative chars-per-token ratio for the prompt estimator. English prose runs ~4 chars/token and code/JSON (common in attachments) ~3, so we use 3: packing never badly under-counts and overflows the model’s true window, while being far less wasteful than the old 2-chars/token guess (which threw away ~half the usable window). Deliberately a portable heuristic, not a model-specific BPE tokenizer — Brain is multi-provider (Ollama/qwen, OpenAI, …) and no single vocabulary is correct for all of them. Swapping in a real tokenizer later only needs to touch this module.