text-normalizer-jp
日本語版 README: README.ja.md
Japanese text normalization for Rust — the neologdn position (pre-tokenizer cleaning) in an NLP pipeline.
- Zero dependencies, no regex, no panic paths. Curated conversion tables are bundled; nothing is pulled in at build time.
- Designed as the stage in front of tokenizers (Lindera / Vibrato / sudachi.rs): cleaning → normalization → masking, then hand off to morphological analysis.
- Composable builder API. Ops run in the order you declare them.
Installation
With the optional html / mask features:
Usage
use Normalizer;
let n = new
.kana_fullwidth // ガギ → ガギ
.ascii_halfwidth // 123ABC → 123ABC
.space_unify
.space_collapse;
assert_eq!;
Or use a preset:
use preset;
let n = neologdn;
assert_eq!;
assert_eq!;
Use cases
- Search-index normalization and matching keys (folding spelling variants, dedupe)
- Corpus cleaning for LLM / RAG pipelines (invisible characters, HTML, URL / e-mail masking)
- Normalizing form input (widths, spaces, kanji numerals in addresses and quantities)
- The cleaning stage in front of Lindera / Vibrato / sudachi.rs tokenization
Coming from Python?
| Python | text-normalizer-jp |
|---|---|
neologdn.normalize(s) |
preset::neologdn().apply(s) |
| mojimoji (zenkaku ⇔ hankaku) | ascii_halfwidth / digits_halfwidth / kana_fullwidth |
| jaconv width conversion | same ops as above — hiragana ⇔ katakana conversion is out of scope |
Ops
| Category | Ops |
|---|---|
| Width | ascii_halfwidth digits_halfwidth kana_fullwidth (dakuten composition: カ+゙→ガ) |
| Space / newline | space_unify space_collapse newline_unify newline_collapse newline_collapse_all trim_lines ja_space_remove ja_space_remove_aggressive |
| Symbols | hyphen_unify (context-aware) wave_dash_unify tilde_remove prolonged_sound_unify prolonged_sound_collapse repeat_suppress(n) |
| Numbers | kanji_numeral_to_arabic (place-value) digit_comma_remove |
| Cleaning | invisible_remove (zero-width, BOM, variation selectors, C0/C1) ascii_lowercase |
| User dictionary | with_replacements(&[(from, to)]) |
HTML (feature html) |
html_detect html_strip |
Masking (feature mask) |
url_mask email_mask mention_remove hashtag_remove phone_mask |
Context-aware dash unification
A dash-like character right after kana becomes ー (a prolonged sound extends
the sound before it); otherwise hyphen-shaped chars (˗ ֊ ‐ ‑ ‒ – ⁃ ⁻ ₋ −)
become - and choonpu-shaped chars (﹣ - ー — ― ─ ━) become ー, with
neologdn-style run collapsing. A blanket replacement turns ス−パ− into
ス-パ-; this op yields スーパー, while 03−1234 still becomes 03-1234
and majika━ stays majikaー.
Place-value kanji numerals
二十三→23, 百五→105, 一万二千→12000, 二〇二五→2025, 〇→0.
Naive per-character replacement produces 2十3 — this is a real parser.
It is pure conversion: place names are not protected (四日市→4日市).
Decide which fields to apply it to (addresses, quantities) at the call site.
Malformed or overflowing runs are left untouched.
neologdn compatibility
preset::neologdn() targets neologdn 0.5.6 (default options). The upstream
test suite is ported in full (tests/neologdn_compat.rs) and passes unmodified;
behavior was also cross-checked against the released PyPI wheel. Intentional
divergences — the kana-context dash rule above, no ¥→\ JIS legacy mapping,
fullwidth orphan dakuten, hiragana dakuten composition — are documented with
rationale in NEOLOGDN_COMPAT.md.
Option mapping: repeat=n → .repeat_suppress(n), remove_space=False →
preset::neologdn_keep_spaces(), tilde='normalize_zenkaku' →
.wave_dash_unify().
Presets
| Preset | Use case |
|---|---|
preset::free_text() |
Safe general cleaning of user-generated text |
preset::search_key() |
Aggressive folding for lookups / dedupe (lowercase, space removal) |
preset::neologdn() |
neologdn 0.5.6-compatible pipeline for NLP preprocessing |
preset::neologdn_keep_spaces() |
Same, with remove_space=False semantics |
Feature flags
html—html_detect/html_strip. Complete-tag detection (a bare<as ina < bis not HTML), entity decoding,<script>/<style>contents dropped.mask— rule-based scanners for URLs, e-mail,@mentions,#hashtags, Japanese phone numbers. For corpus cleaning; not NER-grade PII masking.
Non-goals
- Romaji ⇔ kana transliteration (that is transcription, not normalization)
- Anything requiring morphological analysis (reading estimation, place-name protection)
- Sentence splitting, deduplication (MinHash etc.), NER-based PII masking
- Deep character-form conversion (old kanji forms, IVS) — see yosina for that
- Full NFC/NFD — see unicode-normalization