Crate icu_segmenter

Expand description

🚧 [Experimental] Segment strings by lines, graphemes, word, and sentences.

This module is published as its own crate (icu_segmenter) and as part of the icu crate. See the latter for more details on the ICU4X project.

This module contains segmenter implementation for the following rules.

Line breaker that is compatible with Unicode Standard Annex #14 and CSS properties.
Grapheme cluster breaker, word breaker, and sentence breaker that are compatible with Unicode Standard Annex #29.

🚧 This code is experimental; it may change at any time, in breaking or non-breaking ways, including in SemVer minor releases. It can be enabled with the "experimental" feature of the icu meta-crate. Use with caution. #2259

Examples

Line Break

Segment a string with default options:

 use icu::segmenter::LineBreakSegmenter;

 let segmenter =
     LineBreakSegmenter::try_new_unstable(&icu_testdata::unstable())
         .expect("Data exists");

 let breakpoints: Vec<usize> =
     segmenter.segment_str("Hello World").collect();
 assert_eq!(&breakpoints, &[6, 11]);

See LineBreakSegmenter for more examples.

Grapheme Cluster Break

See GraphemeClusterBreakSegmenter for examples.

Word Break

Segment a string:

 use icu::segmenter::WordBreakSegmenter;

 let segmenter =
     WordBreakSegmenter::try_new_unstable(&icu_testdata::unstable())
         .expect("Data exists");

 let breakpoints: Vec<usize> =
     segmenter.segment_str("Hello World").collect();
 assert_eq!(&breakpoints, &[0, 5, 6, 11]);

See WordBreakSegmenter for more examples.

Sentence Break

See SentenceBreakSegmenter for examples.

Modules

provider

Data provider struct definitions for this ICU4X component.

Structs

GraphemeClusterBreakSegmenter

Segments a string into grapheme clusters.

LineBreakIterator

Implements the Iterator trait over the line break opportunities of the given string. Please see the examples in LineBreakSegmenter for its usages.

LineBreakOptions

Options to tailor line breaking behavior, such as for CSS.

LineBreakSegmenter

Supports loading line break data, and creating line break iterators for different string encodings.

RuleBreakIterator

Implements the Iterator trait over the segmenter break opportunities of the given string.

SentenceBreakSegmenter

Supports loading sentence break data, and creating sentence break iterators for different string encodings.

WordBreakSegmenter

Supports loading word break data, and creating word break iterators for different string encodings.