pub fn needs_multilingual(text: &str) -> boolExpand description
Returns true when text should be routed to the multilingual embedding model
for dense retrieval.
Triggers when >15% of alphabetic characters are non-ASCII-alphabetic.
Denominator is alphabetic-character count (not total characters), so punctuation,
digits, and whitespace do not dilute the signal. This means Müller and
Müller? and Müller!!! all route identically.
Covers: CJK, Cyrillic, Arabic, Devanagari, Hebrew, Thai, accented-Latin (é, ü, ñ, …) and any other non-ASCII script Unicode recognises as alphabetic, without introducing new crate dependencies.
Known limitation: ASCII-only non-English Latin (bonjour le monde,
como estas, ich suche einen buchhalter) is NOT detected and routes to the
primary model. Real language detection for that case requires a dedicated crate
and is tracked as a follow-up to issue #101.