pub fn token_entropy(text: &str) -> f64
Shannon entropy over BPE token IDs (o200k_base). More LLM-relevant than character entropy since LLMs process BPE tokens.