pub const SYMBOL_FACTOR: f32 = 0.5;Expand description
Tokens per ASCII symbol (punctuation) character (default: 0.5)
Symbols like ->, (, { are typically tokenized separately from
identifiers. Counting them at the latin factor would underestimate
code-heavy text; 0.5 per symbol approximates standalone punctuation tokens.