Expand description
Japanese text normalization — the neologdn position (pre-tokenizer cleaning) for Rust.
Zero dependencies, no regex, no panic paths. Designed as the first stage of an NLP pipeline, in front of tokenizers such as Lindera / Vibrato / sudachi.rs: cleaning → normalization → masking, then hand off to morphological analysis.
§Quick start
Build a Normalizer from the ops you need — they run in the order you
declare them — then call apply:
use text_normalizer_jp::Normalizer;
let n = Normalizer::new()
.kana_fullwidth() // ガギ → ガギ
.ascii_halfwidth() // 123ABC → 123ABC
.space_unify()
.space_collapse();
assert_eq!(n.apply("123ABC ガギ"), "123ABC ガギ");A pipeline is built once and applied many times; it is Clone and immutable
after construction. Or start from a preset and keep chaining:
use text_normalizer_jp::preset;
let n = preset::neologdn().repeat_suppress(1);
assert_eq!(n.apply("PRML 副 読 本"), "PRML副読本");§Choosing ops
| Goal | Ops |
|---|---|
| Width folding | ascii_halfwidth, digits_halfwidth, kana_fullwidth |
| Spaces / newlines | space_unify, space_collapse, newline_unify, newline_collapse, newline_collapse_all, trim_lines, ja_space_remove, ja_space_remove_aggressive |
| Symbols | hyphen_unify, wave_dash_unify, tilde_remove, prolonged_sound_unify, prolonged_sound_collapse, repeat_suppress |
| Numbers | kanji_numeral_to_arabic, digit_comma_remove |
| Cleaning | invisible_remove, ascii_lowercase |
| User dictionary | with_replacements |
Order matters. Two rules cover most pitfalls:
- Run
hyphen_unifyand tilde handling beforeascii_halfwidth:-(U+FF0D) and~(U+FF5E) must be classified as dash / tilde while they are still fullwidth. - Run
kana_fullwidthearly, so ops that look at kana context (hyphen_unify,ja_space_remove) see composed fullwidth kana.
§Direct functions
Every op is also a plain fn(&str) -> String in a submodule — width,
space, symbol, number, clean — usable without the builder:
assert_eq!(text_normalizer_jp::number::kanji_numeral_to_arabic("二十三"), "23");§Feature flags (off by default)
html— modulehtml:html_detect/html_strip. Complete-tag detection, entity decoding,<script>/<style>contents dropped.mask— modulemask: URL / e-mail /@mention/#hashtag/ phone masking for corpus cleaning — not NER-grade PII masking.
§neologdn compatibility
preset::neologdn targets neologdn 0.5.6 and passes the upstream test
suite, ported in full. Intentional divergences are listed on the preset and
explained in
NEOLOGDN_COMPAT.md.
§Non-goals
Romaji ⇔ kana transliteration, anything that needs morphological analysis (reading estimation, place-name protection), sentence splitting, NER-based PII masking, old kanji / IVS conversion (see the yosina crate), and full Unicode NFC/NFD (see unicode-normalization).
Modules§
- clean
- Practical cleaning: invisible characters, control characters, ASCII case.
- html
html - HTML detection and stripping (feature
html). - mask
mask - Masking / removal for corpus cleaning (feature
mask): rule-based scanners for URLs, e-mail addresses,@mentions,#hashtags, and Japanese phone numbers — no regex. NER-grade PII (names, addresses) is out of scope by design. - number
- Numeric normalization: place-value kanji numerals, digit-grouping commas.
- preset
- Preset pipelines for common use cases.
- space
- Space and newline normalization, including Japanese-context space removal.
- symbol
- Symbol normalization: dash-like chars, wave dashes / tildes, prolonged sound marks, repeated substrings.
- width
- Width normalization: fullwidth ASCII → halfwidth, halfwidth katakana → fullwidth with dakuten/handakuten composition.
Structs§
- Normalizer
- Composable normalization pipeline. Build with the fluent methods, run with
Normalizer::apply. Ops execute in insertion order.