Struct vibrato::tokenizer::worker::Worker

pub struct Worker<'a> { /* private fields */ }

Provider of a routine for tokenization.

It holds the internal data structures used in tokenization, which can be reused to avoid unnecessary memory reallocation.

Implementations

impl<'a> Worker<'a>

pub fn reset_sentence<S>(&mut self, input: S) -> Result<()>where
    S: AsRef<str>,

Resets the input sentence to be tokenized.

Errors

When the input sentence includes characters more than MAX_SENTENCE_LENGTH, an error will be returned.

pub fn tokenize(&mut self)

Tokenizes the input sentence set in state, returning the result through state.

pub fn num_tokens(&self) -> usize

Gets the number of resultant tokens.

pub fn token(&self, i: usize) -> Token<'_>

Gets the i-th resultant token.

pub const fn token_iter(&'a self) -> TokenIter<'a>

Creates an iterator of resultant tokens.

pub fn init_connid_counter(&mut self)

Initializes a counter to compute occurrence probabilities of connection ids.

pub fn update_connid_counts(&mut self)

Updates frequencies of connection ids at the last tokenization.

Panics

It will panic when Self::init_connid_counter() has never been called.

pub fn compute_connid_probs(&self) -> (Vec<(usize, f64)>, Vec<(usize, f64)>)

Computes the computed occurrence probabilities of connection ids, returning those for left- and right-ids.

Panics

It will panic when Self::init_connid_counter() has never been called.