Skip to main content

Module unicode

Module unicode 

Source
Expand description

Java org.gbif.nameparser.util.UnicodeUtils — ported subset: normalizeQuotes, plus the homoglyph-replacement table (HOMOGLYHPS/containsHomoglyphs/replaceHomoglyphs) used by pipeline::stripandstash’s step 13. decompose/foldToAscii/ replaceSpecialCases/removeNonAscii/replaceNonAscii/unescapeUnicodeChars/ decodeUtf8Garbage/containsDiacritics/findDiacritics are NOT ported: no call site anywhere in this crate’s (now complete) pipeline reaches them, and the full-corpus Rust-vs-Java parity cross-validation (see BENCHMARKS.md/cross-validation.md, 0 mismatches) confirms their absence isn’t a gap.

Functions§

contains_homoglyphs
Java UnicodeUtils.containsHomoglyphs(CharSequence) (UnicodeUtils.java:194-196, delegating to findHomoglyph at :203-215). Returns true if s contains at least one character that is a known homoglyph of a Latin letter. Java’s version short-circuits via a codepoint lower/upper-bound check before the map lookup (an Int2CharMap-specific micro-optimisation); a HashMap<char, _> lookup here is already O(1) average without needing that bound check reproduced.
java_trim
Faithful port of Java String.trim(): strips only leading/trailing chars whose codepoint is <= U+0020 (NOT the full Unicode White_Space set that Rust’s str::trim uses). Use this everywhere the Java source calls .trim().
normalize_quotes
Normalises the many unicode apostrophe / single-quote variants to the ASCII apostrophe (’) and the unicode double-quote variants to the ASCII double quote (“). Author names and quoted/provisional names routinely arrive with curly, prime, modifier-letter, low-9, fullwidth or Windows-1252 quotes; collapsing them to ASCII keeps tokenisation and the parsed output consistent regardless of the input’s quote style.
normalize_spaces
Folds every Unicode space separator (category Zs: NBSP U+00A0, U+1680, U+2000–U+200A, the narrow NBSP U+202F, U+205F, U+3000) to an ASCII space. The parser’s own whitespace handling is ASCII-only (Java parity: String.trim(), Character.isWhitespace, and the (?-u:\s) regexes all skip NBSP), so an NBSP left in place glues words together and hides keywords from every stage — Assimineidae\u{a0}incertae\u{a0}sedis read as a monomial with the author incertae sedis. Runs once on the raw input, before java_trim, so the guards and every later stage see plain spaces. Borrows when there is nothing to fold (the common case).
replace_homoglyphs
Java UnicodeUtils.replaceHomoglyphs(CharSequence, boolean) / its 3-arg sibling (UnicodeUtils.java:239-276), specialised to the only shape any ported call site ever needs: inclHyphens=false and no keep set. pipeline::stripandstash’s step 13 (the sole caller) always passes inclHyphens=false — Unicode hyphen variants are already normalised to ASCII - by the immediately-preceding step 12 (normaliseHyphens), per that Java method’s own comment (“Hyphen homoglyphs are intentionally excluded — those are normalised above already”) — so HYPHEN_HOMOGLYHPS (UnicodeUtils.java:141-167) and the keep-list parameter are both unreachable dead weight for this port’s actual call graph and are not ported. Replaces every character present as a key in [HOMOGLYPHS] with its mapped canonical; every other character passes through unchanged.