Expand description
Java org.gbif.nameparser.util.UnicodeUtils — ported subset: normalizeQuotes, plus
the homoglyph-replacement table (HOMOGLYHPS/containsHomoglyphs/replaceHomoglyphs)
used by pipeline::stripandstash’s step 13. decompose/foldToAscii/
replaceSpecialCases/removeNonAscii/replaceNonAscii/unescapeUnicodeChars/
decodeUtf8Garbage/containsDiacritics/findDiacritics are NOT ported: no call site
anywhere in this crate’s (now complete) pipeline reaches them, and the full-corpus
Rust-vs-Java parity cross-validation (see BENCHMARKS.md/cross-validation.md, 0
mismatches) confirms their absence isn’t a gap.
Functions§
- contains_
homoglyphs - Java
UnicodeUtils.containsHomoglyphs(CharSequence)(UnicodeUtils.java:194-196, delegating tofindHomoglyphat:203-215). Returns true ifscontains at least one character that is a known homoglyph of a Latin letter. Java’s version short-circuits via a codepoint lower/upper-bound check before the map lookup (anInt2CharMap-specific micro-optimisation); aHashMap<char, _>lookup here is already O(1) average without needing that bound check reproduced. - java_
trim - Faithful port of Java
String.trim(): strips only leading/trailing chars whose codepoint is <= U+0020 (NOT the full Unicode White_Space set that Rust’s str::trim uses). Use this everywhere the Java source calls.trim(). - normalize_
quotes - Normalises the many unicode apostrophe / single-quote variants to the ASCII apostrophe (’) and the unicode double-quote variants to the ASCII double quote (“). Author names and quoted/provisional names routinely arrive with curly, prime, modifier-letter, low-9, fullwidth or Windows-1252 quotes; collapsing them to ASCII keeps tokenisation and the parsed output consistent regardless of the input’s quote style.
- normalize_
spaces - Folds every Unicode space separator (category
Zs: NBSP U+00A0, U+1680, U+2000–U+200A, the narrow NBSP U+202F, U+205F, U+3000) to an ASCII space. The parser’s own whitespace handling is ASCII-only (Java parity:String.trim(),Character.isWhitespace, and the(?-u:\s)regexes all skip NBSP), so an NBSP left in place glues words together and hides keywords from every stage —Assimineidae\u{a0}incertae\u{a0}sedisread as a monomial with the authorincertae sedis. Runs once on the raw input, beforejava_trim, so the guards and every later stage see plain spaces. Borrows when there is nothing to fold (the common case). - replace_
homoglyphs - Java
UnicodeUtils.replaceHomoglyphs(CharSequence, boolean)/ its 3-arg sibling (UnicodeUtils.java:239-276), specialised to the only shape any ported call site ever needs:inclHyphens=falseand nokeepset.pipeline::stripandstash’s step 13 (the sole caller) always passesinclHyphens=false— Unicode hyphen variants are already normalised to ASCII-by the immediately-preceding step 12 (normaliseHyphens), per that Java method’s own comment (“Hyphen homoglyphs are intentionally excluded — those are normalised above already”) — soHYPHEN_HOMOGLYHPS(UnicodeUtils.java:141-167) and thekeep-list parameter are both unreachable dead weight for this port’s actual call graph and are not ported. Replaces every character present as a key in [HOMOGLYPHS] with its mapped canonical; every other character passes through unchanged.