text-normalizer-jp 0.1.0

Japanese text normalization: width, kana, spaces, symbols, kanji numerals, neologdn-compatible preset. Zero dependencies, no regex, no panics. Preprocessing for Lindera / Vibrato / sudachi.rs.
Documentation
  • Coverage
  • 100%
    72 out of 72 items documented18 out of 72 items with examples
  • Size
  • Source code size: 103.7 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 530.9 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 2s Average build duration of successful builds.
  • all releases: 2s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • uiuifree/rust-text-normalizer-jp
    0 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • uiuifree

text-normalizer-jp

crates.io docs.rs license

日本語版 README: README.ja.md

Japanese text normalization for Rust — the neologdn position (pre-tokenizer cleaning) in an NLP pipeline.

  • Zero dependencies, no regex, no panic paths. Curated conversion tables are bundled; nothing is pulled in at build time.
  • Designed as the stage in front of tokenizers (Lindera / Vibrato / sudachi.rs): cleaning → normalization → masking, then hand off to morphological analysis.
  • Composable builder API. Ops run in the order you declare them.

Installation

cargo add text-normalizer-jp

With the optional html / mask features:

cargo add text-normalizer-jp --features html,mask

Usage

use text_normalizer_jp::Normalizer;

let n = Normalizer::new()
    .kana_fullwidth()   // ガギ → ガギ
    .ascii_halfwidth()  // 123ABC → 123ABC
    .space_unify()
    .space_collapse();
assert_eq!(n.apply("123ABC ガギ"), "123ABC ガギ");

Or use a preset:

use text_normalizer_jp::preset;

let n = preset::neologdn();
assert_eq!(n.apply("PRML  副 読 本"), "PRML副読本");
assert_eq!(n.apply("Coding the Matrix"), "Coding the Matrix");

Use cases

  • Search-index normalization and matching keys (folding spelling variants, dedupe)
  • Corpus cleaning for LLM / RAG pipelines (invisible characters, HTML, URL / e-mail masking)
  • Normalizing form input (widths, spaces, kanji numerals in addresses and quantities)
  • The cleaning stage in front of Lindera / Vibrato / sudachi.rs tokenization

Coming from Python?

Python text-normalizer-jp
neologdn.normalize(s) preset::neologdn().apply(s)
mojimoji (zenkaku ⇔ hankaku) ascii_halfwidth / digits_halfwidth / kana_fullwidth
jaconv width conversion same ops as above — hiragana ⇔ katakana conversion is out of scope

Ops

Category Ops
Width ascii_halfwidth digits_halfwidth kana_fullwidth (dakuten composition: カ+゙→ガ)
Space / newline space_unify space_collapse newline_unify newline_collapse newline_collapse_all trim_lines ja_space_remove ja_space_remove_aggressive
Symbols hyphen_unify (context-aware) wave_dash_unify tilde_remove prolonged_sound_unify prolonged_sound_collapse repeat_suppress(n)
Numbers kanji_numeral_to_arabic (place-value) digit_comma_remove
Cleaning invisible_remove (zero-width, BOM, variation selectors, C0/C1) ascii_lowercase
User dictionary with_replacements(&[(from, to)])
HTML (feature html) html_detect html_strip
Masking (feature mask) url_mask email_mask mention_remove hashtag_remove phone_mask

Context-aware dash unification

A dash-like character right after kana becomes ー (a prolonged sound extends the sound before it); otherwise hyphen-shaped chars (˗ ֊ ‐ ‑ ‒ – ⁃ ⁻ ₋ −) become - and choonpu-shaped chars (﹣ - ー — ― ─ ━) become ー, with neologdn-style run collapsing. A blanket replacement turns ス−パ− into ス-パ-; this op yields スーパー, while 03−1234 still becomes 03-1234 and majika━ stays majikaー.

Place-value kanji numerals

二十三→23, 百五→105, 一万二千→12000, 二〇二五→2025, 〇→0. Naive per-character replacement produces 2十3 — this is a real parser. It is pure conversion: place names are not protected (四日市→4日市). Decide which fields to apply it to (addresses, quantities) at the call site. Malformed or overflowing runs are left untouched.

neologdn compatibility

preset::neologdn() targets neologdn 0.5.6 (default options). The upstream test suite is ported in full (tests/neologdn_compat.rs) and passes unmodified; behavior was also cross-checked against the released PyPI wheel. Intentional divergences — the kana-context dash rule above, no ¥→\ JIS legacy mapping, fullwidth orphan dakuten, hiragana dakuten composition — are documented with rationale in NEOLOGDN_COMPAT.md.

Option mapping: repeat=n → .repeat_suppress(n), remove_space=False → preset::neologdn_keep_spaces(), tilde='normalize_zenkaku' → .wave_dash_unify().

Presets

Preset Use case
preset::free_text() Safe general cleaning of user-generated text
preset::search_key() Aggressive folding for lookups / dedupe (lowercase, space removal)
preset::neologdn() neologdn 0.5.6-compatible pipeline for NLP preprocessing
preset::neologdn_keep_spaces() Same, with remove_space=False semantics

Feature flags

  • html — html_detect / html_strip. Complete-tag detection (a bare < as in a < b is not HTML), entity decoding, <script>/<style> contents dropped.
  • mask — rule-based scanners for URLs, e-mail, @mentions, #hashtags, Japanese phone numbers. For corpus cleaning; not NER-grade PII masking.

Non-goals

  • Romaji ⇔ kana transliteration (that is transcription, not normalization)
  • Anything requiring morphological analysis (reading estimation, place-name protection)
  • Sentence splitting, deduplication (MinHash etc.), NER-based PII masking
  • Deep character-form conversion (old kanji forms, IVS) — see yosina for that
  • Full NFC/NFD — see unicode-normalization

License

MIT