Struct Segmenter

pub struct Segmenter { /* private fields */ }

Central data structure used to calculate word probabilities

Implementations

impl Segmenter

pub fn new<U, B>(unigrams: U, bigrams: B) -> Self

where U: IntoIterator<Item = (String, f64)>, B: IntoIterator<Item = ((String, String), f64)>,

Create Segmenter from the given unigram and bigram counts.

Note: the String types used in this API are defined in the smartstring crate. Any &str or String can be converted into the String used here by calling into() on it.

pub fn segment<'a>( &self, input: &str, search: &'a mut Search, ) -> Result<Segments<'a>, InvalidCharacter>

Segment the text in input

Requires that the input text consists of lowercase ASCII characters only. Otherwise, returns Err(InvalidCharacter). The search parameter contains caches that are used segmentation; passing it in allows the callers to reuse the cache allocations.

pub fn score_sentence<'a>( &self, words: impl Iterator<Item = &'a str>, ) -> Option<f64>

Returns the sentence’s score

Returns the relative probability for the given sentence in the the corpus represented by this Segmenter. Will return None iff given an empty iterator argument.

pub fn set_limit(&mut self, limit: usize)

Customize the word length limit