Struct instant_segment::Segmenter

pub struct Segmenter { /* private fields */ }

Central data structure used to calculate word probabilities

Implementations

impl Segmenter

pub fn from_iters<U, B>(unigrams: U, bigrams: B) -> Selfwhere
    U: Iterator<Item = (String, f64)>,
    B: Iterator<Item = ((String, String), f64)>,

Create Segmenter from the given iterators

Note: the String types used in this API are defined in the smartstring crate. Any &str or String can be converted into the String used here by calling into() on it.

pub fn from_maps(
    unigrams: HashMap<String, f64, RandomState>,
    bigrams: HashMap<(String, String), f64, RandomState>
) -> Self

Create Segmenter from the given hashmaps (using ahash)

Note: the String types used in this API are defined in the smartstring crate. Any &str or String can be converted into the String used here by calling into() on it. The HashMap type here refers to std::collections::HashMap parametrized with the ahash::RandomState.

pub fn segment<'a>(
    &self,
    input: &str,
    search: &'a mut Search
) -> Result<impl Iterator<Item = &'a str> + ExactSizeIterator, InvalidCharacter>

Segment the text in input

Requires that the input text consists of lowercase ASCII characters only. Otherwise, returns Err(InvalidCharacter). The search parameter contains caches that are used segmentation; passing it in allows the callers to reuse the cache allocations.

pub fn score_sentence<'a>(
    &self,
    words: impl Iterator<Item = &'a str>
) -> Option<f64>

Returns the sentence’s score

Returns the relative probability for the given sentence in the the corpus represented by this Segmenter. Will return None iff given an empty iterator argument.

pub fn set_limit(&mut self, limit: usize)

Customize the word length limit