Struct general_sam::utils::tokenize::GreedyTokenizer

pub struct GreedyTokenizer<'s, TransTable: TransitionTable, TokenIDType: Clone + Default + PartialEq> { /* private fields */ }

Available on crate feature utils only.

Greedy tokenizer with a general suffix automaton of the vocabulary.

Assuming that the input length is $n$, the maximum word length is $l$, and querying transitions in the trie takes $\mathcal{O}\left(\log{\Sigma}\right)$ time, then the overall time complexity of this implementation is $\mathcal{O}\left( n \cdot \left( \log{l} + \log{\Sigma} \right) \right)$.

The main optimization is to store suffix-wise information with persistent ropes. For each suffix in a state of the suffix automaton, the longest word matching the prefix of the suffix is stored in the rope. And the information stored in a state will be further merged in the ropes of its successors.

Implementations

impl<'s, TransTable: TransitionTable> GreedyTokenizer<'s, TransTable, TrieNodeID>

pub fn build_from_trie<'t, TT: TransitionTable<KeyType = TransTable::KeyType>>( sam: &'s GeneralSAM<TransTable>, trie_state: TrieState<'t, TT> ) -> Self

Available on crate feature trie only.

impl<'s, TransTable: TransitionTable, TokenIDType: Clone + Default + PartialEq> GreedyTokenizer<'s, TransTable, TokenIDType>

pub fn build<TN: TrieNodeAlike<InnerType = TransTable::KeyType>, F: FnMut(&TN) -> TokenIDType>( sam: &'s GeneralSAM<TransTable>, trie_node: TN, f: F ) -> Self

pub fn tokenize<Iter: Iterator<Item = TransTable::KeyType>>( &self, iter: Iter, unk_token_id: &TokenIDType ) -> Vec<(TokenIDType, usize)>

Trait Implementations

impl<'s, TransTable: Clone + TransitionTable, TokenIDType: Clone + Clone + Default + PartialEq> Clone for GreedyTokenizer<'s, TransTable, TokenIDType>

fn clone(&self) -> GreedyTokenizer<'s, TransTable, TokenIDType>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'s, TransTable: Debug + TransitionTable, TokenIDType: Debug + Clone + Default + PartialEq> Debug for GreedyTokenizer<'s, TransTable, TokenIDType>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.