Skip to main content

Normalizer

Struct Normalizer 

Source
pub struct Normalizer { /* private fields */ }
Expand description

Composable normalization pipeline. Build with the fluent methods, run with Normalizer::apply. Ops execute in insertion order.

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().ascii_halfwidth().space_collapse();
assert_eq!(n.apply("Rust  入門"), "Rust 入門");

Implementations§

Source§

impl Normalizer

Source

pub fn new() -> Self

Empty pipeline — apply returns input unchanged until ops are added with the builder methods.

Source

pub fn ascii_halfwidth(self) -> Self

Fullwidth ASCII (U+FF01–U+FF5E) → halfwidth. Note: ~ (U+FF5E) becomes ~; run Self::wave_dash_unify or Self::tilde_remove first if you want different tilde handling.

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().ascii_halfwidth();
assert_eq!(n.apply("123ABC()"), "123ABC()");
Source

pub fn digits_halfwidth(self) -> Self

Fullwidth digits only (U+FF10–U+FF19) → halfwidth. 123 → 123.

Source

pub fn kana_fullwidth(self) -> Self

Halfwidth katakana → fullwidth, composing dakuten/handakuten. Fullwidth and combining marks compose with any kana base too — see width::kana_fullwidth.

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().kana_fullwidth();
assert_eq!(n.apply("ガギヴ"), "ガギヴ");
assert_eq!(n.apply("う゛"), "ゔ");
Source

pub fn space_unify(self) -> Self

Ideographic space (U+3000) and tab → single halfwidth space.

Source

pub fn space_collapse(self) -> Self

Runs of spaces/tabs/ideographic spaces → one halfwidth space.

Source

pub fn newline_unify(self) -> Self

CRLF / CR → LF.

Source

pub fn newline_collapse(self) -> Self

Runs of 3+ newlines → 2 (paragraphs kept). newline_collapse_all for runs → 1.

Source

pub fn newline_collapse_all(self) -> Self

Any run of newlines → a single newline.

Source

pub fn trim_lines(self) -> Self

Strip leading/trailing spaces, tabs and ideographic spaces on every line.

Source

pub fn ja_space_remove(self) -> Self

Remove spaces between two Japanese characters; a Latin boundary keeps its space.

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().ja_space_remove();
assert_eq!(n.apply("検索 エンジン"), "検索エンジン");
assert_eq!(n.apply("Rust 入門"), "Rust 入門");
Source

pub fn ja_space_remove_aggressive(self) -> Self

neologdn-compatible aggressive variant: remove a space when at least one side is Japanese. Includes the neologdn quirks documented on space::ja_space_remove.

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().ja_space_remove_aggressive();
assert_eq!(n.apply("PRML 副読本"), "PRML副読本");
assert_eq!(n.apply("Coding the Matrix"), "Coding the Matrix");
Source

pub fn hyphen_unify(self) -> Self

Dash-like chars: after kana → ー; otherwise hyphen-shaped (˗ ֊ ‐ ‑ ‒ – ⁃ ⁻ ₋ −) → - and choonpu-shaped (﹣ - ー — ― ─ ━) → ー, with neologdn-style run collapsing. See symbol::hyphen_unify.

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().hyphen_unify();
assert_eq!(n.apply("ス\u{2212}パ\u{2212}"), "スーパー"); // kana context → ー
assert_eq!(n.apply("03\u{2212}1234"), "03-1234");        // elsewhere → -
Source

pub fn wave_dash_unify(self) -> Self

Tilde-like chars (~ ~ ∼ ∾ 〰) → 〜 (neologdn tilde='normalize_zenkaku'). Mutually exclusive in spirit with Self::tilde_remove; pick one.

Source

pub fn tilde_remove(self) -> Self

Remove tilde-like chars (~ 〜 ~ ∼ ∾ 〰) — the neologdn behavior.

Source

pub fn prolonged_sound_unify(self) -> Self

Prolonged-sound-like chars (ー ˗) → ー. Runs before collapse.

Source

pub fn prolonged_sound_collapse(self) -> Self

Runs of ー → one ー. スーーパーー → スーパー.

Source

pub fn repeat_suppress(self, max: usize) -> Self

Suppress repeated substrings to at most max consecutive occurrences — neologdn’s repeat behavior, including multi-char units. Substring length is capped at 8 chars (neologdn default); use symbol::shorten_repeat directly for other caps. max == 0 is a no-op.

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().repeat_suppress(2);
assert_eq!(n.apply("かわいいいいい"), "かわいい");

let n = Normalizer::new().repeat_suppress(1);
assert_eq!(n.apply("無駄無駄無駄無駄ァ"), "無駄ァ");
Source

pub fn digit_comma_remove(self) -> Self

Remove , / , between ASCII digits. 1,234 → 1234.

Source

pub fn kanji_numeral_to_arabic(self) -> Self

Place-value kanji numerals → Arabic. Applies everywhere it matches — where to apply it (address fields etc.) is the caller’s responsibility: place names are not protected (四日市 → 4日市).

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().kanji_numeral_to_arabic();
assert_eq!(n.apply("第三章の十五ページ"), "第3章の15ページ");
assert_eq!(n.apply("一万二千"), "12000");
assert_eq!(n.apply("二〇二五年"), "2025年");
Source

pub fn invisible_remove(self) -> Self

Remove zero-width chars, BOM, variation selectors, soft hyphen and C0/C1 controls (keeps \n and \t); NBSP → space.

Source

pub fn ascii_lowercase(self) -> Self

ASCII A–Z → a–z. Non-ASCII untouched.

Source

pub fn with_replacements(self, pairs: &[(&str, &str)]) -> Self

User dictionary: apply (from, to) pairs in declaration order. Domain rules belong here, not in the core.

use text_normalizer_jp::Normalizer;

let n = Normalizer::new().with_replacements(&[("株式会会社", "株式会社")]);
assert_eq!(n.apply("株式会会社ロクゼロ"), "株式会社ロクゼロ");
Source

pub fn html_strip(self) -> Self

Available on crate feature html only.

Strip HTML tags and decode entities. See html::html_strip.

Source

pub fn url_mask(self, placeholder: Option<&str>) -> Self

Available on crate feature mask only.

Replace URLs with placeholder, or remove them when None.

Source

pub fn email_mask(self, placeholder: &str) -> Self

Available on crate feature mask only.

Replace e-mail addresses with placeholder.

Source

pub fn mention_remove(self) -> Self

Available on crate feature mask only.

Remove @mention tokens.

Source

pub fn hashtag_remove(self) -> Self

Available on crate feature mask only.

Remove #hashtag tokens.

Source

pub fn phone_mask(self, placeholder: &str) -> Self

Available on crate feature mask only.

Replace phone-number-shaped digit runs with placeholder.

Source

pub fn apply(&self, input: &str) -> String

Run the pipeline over input, applying ops in the order they were added. Never panics; an empty pipeline returns the input unchanged.

Trait Implementations§

Source§

impl Clone for Normalizer

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for Normalizer

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for Normalizer

Source§

fn default() -> Self

Returns the “default value” for a type. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.