Struct icu::segmenter::WordSegmenter
source · pub struct WordSegmenter { /* private fields */ }
Expand description
Supports loading word break data, and creating word break iterators for different string encodings.
Examples
Segment a string:
use icu_segmenter::WordSegmenter;
let segmenter =
WordSegmenter::try_new_auto_unstable(&icu_testdata::unstable())
.expect("Data exists");
let breakpoints: Vec<usize> =
segmenter.segment_str("Hello World").collect();
assert_eq!(&breakpoints, &[0, 5, 6, 11]);
Segment a Latin1 byte string:
use icu_segmenter::WordSegmenter;
let segmenter =
WordSegmenter::try_new_auto_unstable(&icu_testdata::unstable())
.expect("Data exists");
let breakpoints: Vec<usize> =
segmenter.segment_latin1(b"Hello World").collect();
assert_eq!(&breakpoints, &[0, 5, 6, 11]);
Successive boundaries can be used to retrieve the segments. In particular, the first boundary is always 0, and the last one is the length of the segmented text in code units.
use itertools::Itertools;
let text = "Mark’d ye his words?";
let segments: Vec<&str> = segmenter
.segment_str(text)
.tuple_windows()
.map(|(i, j)| &text[i..j])
.collect();
assert_eq!(&segments, &["Mark’d", " ", "ye", " ", "his", " ", "words", "?"]);
Not all segments delimited by word boundaries are words; some are interword
segments such as spaces and punctuation.
The WordBreakIterator::word_type()
of a boundary can be used to
classify the preceding segment.
let words: Vec<&str> = {
let mut it = segmenter.segment_str(text);
std::iter::from_fn(move || it.next().map(|i| (i, it.word_type())))
.tuple_windows()
.filter(|(_, (_, status))| *status == WordType::Letter)
.map(|((i, _), (j, _))| &text[i..j])
.collect()
};
assert_eq!(&words, &["Mark’d", "ye", "his", "words"]);
Implementations§
source§impl WordSegmenter
impl WordSegmenter
sourcepub fn try_new_auto_unstable<D>(
provider: &D
) -> Result<WordSegmenter, SegmenterError>where
D: DataProvider<WordBreakDataV1Marker> + DataProvider<DictionaryForWordOnlyAutoV1Marker> + DataProvider<LstmForWordLineAutoV1Marker> + DataProvider<GraphemeClusterBreakDataV1Marker> + ?Sized,
pub fn try_new_auto_unstable<D>( provider: &D ) -> Result<WordSegmenter, SegmenterError>where D: DataProvider<WordBreakDataV1Marker> + DataProvider<DictionaryForWordOnlyAutoV1Marker> + DataProvider<LstmForWordLineAutoV1Marker> + DataProvider<GraphemeClusterBreakDataV1Marker> + ?Sized,
Constructs a WordSegmenter
with an invariant locale and the best available data for
complex scripts (Chinese, Japanese, Khmer, Lao, Myanmar, and Thai).
The current behavior, which is subject to change, is to use the LSTM model when available and the dictionary model for Chinese and Japanese.
Examples
Behavior with complex scripts:
use icu::segmenter::WordSegmenter;
let th_str = "ทุกสองสัปดาห์";
let ja_str = "こんにちは世界";
let segmenter =
WordSegmenter::try_new_auto_unstable(&icu_testdata::unstable())
.unwrap();
let th_bps = segmenter.segment_str(th_str).collect::<Vec<_>>();
let ja_bps = segmenter.segment_str(ja_str).collect::<Vec<_>>();
assert_eq!(th_bps, [0, 9, 18, 39]);
assert_eq!(ja_bps, [0, 15, 21]);
sourcepub fn try_new_auto_with_any_provider(
provider: &impl AnyProvider
) -> Result<WordSegmenter, SegmenterError>
pub fn try_new_auto_with_any_provider( provider: &impl AnyProvider ) -> Result<WordSegmenter, SegmenterError>
Creates a new instance using an AnyProvider
.
For details on the behavior of this function, see: Self::try_new_auto_unstable
sourcepub fn try_new_auto_with_buffer_provider(
provider: &impl BufferProvider
) -> Result<WordSegmenter, SegmenterError>
pub fn try_new_auto_with_buffer_provider( provider: &impl BufferProvider ) -> Result<WordSegmenter, SegmenterError>
✨ Enabled with the "serde"
feature.
Creates a new instance using a BufferProvider
.
For details on the behavior of this function, see: Self::try_new_auto_unstable
sourcepub fn try_new_lstm_unstable<D>(
provider: &D
) -> Result<WordSegmenter, SegmenterError>where
D: DataProvider<WordBreakDataV1Marker> + DataProvider<LstmForWordLineAutoV1Marker> + DataProvider<GraphemeClusterBreakDataV1Marker> + ?Sized,
pub fn try_new_lstm_unstable<D>( provider: &D ) -> Result<WordSegmenter, SegmenterError>where D: DataProvider<WordBreakDataV1Marker> + DataProvider<LstmForWordLineAutoV1Marker> + DataProvider<GraphemeClusterBreakDataV1Marker> + ?Sized,
Constructs a WordSegmenter
with an invariant locale and LSTM data for
complex scripts (Burmese, Khmer, Lao, and Thai).
The LSTM, or Long Term Short Memory, is a machine learning model. It is smaller than the full dictionary but more expensive during segmentation (inference).
Warning: there is not currently an LSTM model for Chinese or Japanese, so the WordSegmenter
created by this function will have unexpected behavior in spans of those scripts.
Examples
Behavior with complex scripts:
use icu::segmenter::WordSegmenter;
let th_str = "ทุกสองสัปดาห์";
let ja_str = "こんにちは世界";
let segmenter =
WordSegmenter::try_new_lstm_unstable(&icu_testdata::unstable())
.unwrap();
let th_bps = segmenter.segment_str(th_str).collect::<Vec<_>>();
let ja_bps = segmenter.segment_str(ja_str).collect::<Vec<_>>();
assert_eq!(th_bps, [0, 9, 18, 39]);
// Note: We aren't able to find a suitable breakpoint in Chinese/Japanese.
assert_eq!(ja_bps, [0, 21]);
sourcepub fn try_new_lstm_with_any_provider(
provider: &impl AnyProvider
) -> Result<WordSegmenter, SegmenterError>
pub fn try_new_lstm_with_any_provider( provider: &impl AnyProvider ) -> Result<WordSegmenter, SegmenterError>
Creates a new instance using an AnyProvider
.
For details on the behavior of this function, see: Self::try_new_lstm_unstable
sourcepub fn try_new_lstm_with_buffer_provider(
provider: &impl BufferProvider
) -> Result<WordSegmenter, SegmenterError>
pub fn try_new_lstm_with_buffer_provider( provider: &impl BufferProvider ) -> Result<WordSegmenter, SegmenterError>
✨ Enabled with the "serde"
feature.
Creates a new instance using a BufferProvider
.
For details on the behavior of this function, see: Self::try_new_lstm_unstable
sourcepub fn try_new_dictionary_unstable<D>(
provider: &D
) -> Result<WordSegmenter, SegmenterError>where
D: DataProvider<WordBreakDataV1Marker> + DataProvider<DictionaryForWordOnlyAutoV1Marker> + DataProvider<DictionaryForWordLineExtendedV1Marker> + DataProvider<GraphemeClusterBreakDataV1Marker> + ?Sized,
pub fn try_new_dictionary_unstable<D>( provider: &D ) -> Result<WordSegmenter, SegmenterError>where D: DataProvider<WordBreakDataV1Marker> + DataProvider<DictionaryForWordOnlyAutoV1Marker> + DataProvider<DictionaryForWordLineExtendedV1Marker> + DataProvider<GraphemeClusterBreakDataV1Marker> + ?Sized,
Construct a WordSegmenter
with an invariant locale and dictionary data for
complex scripts (Chinese, Japanese, Khmer, Lao, Myanmar, and Thai).
The dictionary model uses a list of words to determine appropriate breakpoints. It is faster than the LSTM model but requires more data.
Examples
Behavior with complex scripts:
use icu::segmenter::WordSegmenter;
let th_str = "ทุกสองสัปดาห์";
let ja_str = "こんにちは世界";
let segmenter =
WordSegmenter::try_new_dictionary_unstable(&icu_testdata::unstable())
.unwrap();
let th_bps = segmenter.segment_str(th_str).collect::<Vec<_>>();
let ja_bps = segmenter.segment_str(ja_str).collect::<Vec<_>>();
assert_eq!(th_bps, [0, 9, 18, 39]);
assert_eq!(ja_bps, [0, 15, 21]);
sourcepub fn try_new_dictionary_with_any_provider(
provider: &impl AnyProvider
) -> Result<WordSegmenter, SegmenterError>
pub fn try_new_dictionary_with_any_provider( provider: &impl AnyProvider ) -> Result<WordSegmenter, SegmenterError>
Creates a new instance using an AnyProvider
.
For details on the behavior of this function, see: Self::try_new_dictionary_unstable
sourcepub fn try_new_dictionary_with_buffer_provider(
provider: &impl BufferProvider
) -> Result<WordSegmenter, SegmenterError>
pub fn try_new_dictionary_with_buffer_provider( provider: &impl BufferProvider ) -> Result<WordSegmenter, SegmenterError>
✨ Enabled with the "serde"
feature.
Creates a new instance using a BufferProvider
.
For details on the behavior of this function, see: Self::try_new_dictionary_unstable
sourcepub fn segment_str<'l, 's>(
&'l self,
input: &'s str
) -> WordBreakIterator<'l, 's, WordBreakTypeUtf8> ⓘ
pub fn segment_str<'l, 's>( &'l self, input: &'s str ) -> WordBreakIterator<'l, 's, WordBreakTypeUtf8> ⓘ
Creates a word break iterator for an str
(a UTF-8 string).
There are always breakpoints at 0 and the string length, or only at 0 for the empty string.
sourcepub fn segment_utf8<'l, 's>(
&'l self,
input: &'s [u8]
) -> WordBreakIterator<'l, 's, WordBreakTypePotentiallyIllFormedUtf8> ⓘ
pub fn segment_utf8<'l, 's>( &'l self, input: &'s [u8] ) -> WordBreakIterator<'l, 's, WordBreakTypePotentiallyIllFormedUtf8> ⓘ
Creates a word break iterator for a potentially ill-formed UTF8 string
Invalid characters are treated as REPLACEMENT CHARACTER
There are always breakpoints at 0 and the string length, or only at 0 for the empty string.
sourcepub fn segment_latin1<'l, 's>(
&'l self,
input: &'s [u8]
) -> WordBreakIterator<'l, 's, RuleBreakTypeLatin1> ⓘ
pub fn segment_latin1<'l, 's>( &'l self, input: &'s [u8] ) -> WordBreakIterator<'l, 's, RuleBreakTypeLatin1> ⓘ
Creates a word break iterator for a Latin-1 (8-bit) string.
There are always breakpoints at 0 and the string length, or only at 0 for the empty string.
sourcepub fn segment_utf16<'l, 's>(
&'l self,
input: &'s [u16]
) -> WordBreakIterator<'l, 's, WordBreakTypeUtf16> ⓘ
pub fn segment_utf16<'l, 's>( &'l self, input: &'s [u16] ) -> WordBreakIterator<'l, 's, WordBreakTypeUtf16> ⓘ
Creates a word break iterator for a UTF-16 string.
There are always breakpoints at 0 and the string length, or only at 0 for the empty string.