Skip to main content

Crate text_normalizer_jp

Crate text_normalizer_jp 

Source
Expand description

Japanese text normalization — the neologdn position (pre-tokenizer cleaning) for Rust.

Zero dependencies, no regex, no panic paths. Designed as the first stage of an NLP pipeline, in front of tokenizers such as Lindera / Vibrato / sudachi.rs: cleaning → normalization → masking, then hand off to morphological analysis.

§Quick start

Build a Normalizer from the ops you need — they run in the order you declare them — then call apply:

use text_normalizer_jp::Normalizer;

let n = Normalizer::new()
    .kana_fullwidth()   // ガギ → ガギ
    .ascii_halfwidth()  // 123ABC → 123ABC
    .space_unify()
    .space_collapse();
assert_eq!(n.apply("123ABC ガギ"), "123ABC ガギ");

A pipeline is built once and applied many times; it is Clone and immutable after construction. Or start from a preset and keep chaining:

use text_normalizer_jp::preset;

let n = preset::neologdn().repeat_suppress(1);
assert_eq!(n.apply("PRML  副 読 本"), "PRML副読本");

§Choosing ops

Order matters. Two rules cover most pitfalls:

§Direct functions

Every op is also a plain fn(&str) -> String in a submodule — width, space, symbol, number, clean — usable without the builder:

assert_eq!(text_normalizer_jp::number::kanji_numeral_to_arabic("二十三"), "23");

§Feature flags (off by default)

  • html — module html: html_detect / html_strip. Complete-tag detection, entity decoding, <script>/<style> contents dropped.
  • mask — module mask: URL / e-mail / @mention / #hashtag / phone masking for corpus cleaning — not NER-grade PII masking.

§neologdn compatibility

preset::neologdn targets neologdn 0.5.6 and passes the upstream test suite, ported in full. Intentional divergences are listed on the preset and explained in NEOLOGDN_COMPAT.md.

§Non-goals

Romaji ⇔ kana transliteration, anything that needs morphological analysis (reading estimation, place-name protection), sentence splitting, NER-based PII masking, old kanji / IVS conversion (see the yosina crate), and full Unicode NFC/NFD (see unicode-normalization).

Modules§

clean
Practical cleaning: invisible characters, control characters, ASCII case.
htmlhtml
HTML detection and stripping (feature html).
maskmask
Masking / removal for corpus cleaning (feature mask): rule-based scanners for URLs, e-mail addresses, @mentions, #hashtags, and Japanese phone numbers — no regex. NER-grade PII (names, addresses) is out of scope by design.
number
Numeric normalization: place-value kanji numerals, digit-grouping commas.
preset
Preset pipelines for common use cases.
space
Space and newline normalization, including Japanese-context space removal.
symbol
Symbol normalization: dash-like chars, wave dashes / tildes, prolonged sound marks, repeated substrings.
width
Width normalization: fullwidth ASCII → halfwidth, halfwidth katakana → fullwidth with dakuten/handakuten composition.

Structs§

Normalizer
Composable normalization pipeline. Build with the fluent methods, run with Normalizer::apply. Ops execute in insertion order.