Skip to main content

detect_language

sdsforge_core::language

Function detect_language 

Source 
pub fn detect_language(text: &str) -> Language
Expand description

Heuristically detect the language of an SDS document from its extracted text.

Detection order: 0. Fewer than 30 non-whitespace characters → Language::Japanese (default, not enough text)

  1. Hiragana or katakana present → Language::Japanese
  2. No CJK ideographs AND text is substantially Latin (≥ 30 % of meaningful chars are ASCII printable a-z/A-Z/0-9) → Language::English This avoids misclassifying Japanese PDFs whose pdftotext output contains only garbage ASCII (e.g. garbled CID/Shift-JIS font metrics) as English.
  3. Fewer than 20 CJK characters → Language::Japanese (not enough CJK to distinguish)
  4. Traditional-Chinese-only characters outnumber simplified-only → Language::ChineseTraditional
  5. Otherwise → Language::ChineseSimplified

Works on as little as ~200 characters of text. No LLM or network call required.