Struct vibrato::tokenizer::worker::Worker

pub struct Worker<'t> { /* private fields */ }

Provider of a routine for tokenization.

It holds the internal data structures used in tokenization, which can be reused to avoid unnecessary memory reallocation.

Implementations

impl<'t> Worker<'t>

pub fn reset_sentence<S>(&mut self, input: S)where S: AsRef<str>,

Resets the input sentence to be tokenized.

pub fn tokenize(&mut self)

Tokenizes the input sentence set in state, returning the result through state.

pub fn num_tokens(&self) -> usize

Gets the number of resultant tokens.

pub fn token<'w>(&'w self, i: usize) -> Token<'w, 't>

Gets the i-th resultant token.

pub const fn token_iter<'w>(&'w self) -> TokenIter<'w, 't>

Creates an iterator of resultant tokens.

pub fn init_connid_counter(&mut self)

Initializes a counter to compute occurrence probabilities of connection ids.

pub fn update_connid_counts(&mut self)

Updates frequencies of connection ids at the last tokenization.

Panics

It will panic when Self::init_connid_counter() has never been called.

pub fn compute_connid_probs(&self) -> (Vec<(usize, f64)>, Vec<(usize, f64)>)

Computes the computed occurrence probabilities of connection ids, returning those for left- and right-ids.

Panics

It will panic when Self::init_connid_counter() has never been called.