Struct TokenTrie 

pub struct TokenTrie { /* private fields */ }

Fast token lookup trie. All token types for a language are inserted into one trie.

Debug prints only the process mask (the full trie is too large to dump).

Implementations

impl TokenTrie

pub fn new() -> Self

pub fn insert(&mut self, pattern: &[u8], token_match: TokenMatch)

pub fn match_at(&self, content: &[u8], pos: usize) -> Option<TokenMatch>

Try to match a token at position pos. Returns longest match (greedy).

pub fn process_mask(&self) -> u8

Trait Implementations

impl Debug for TokenTrie

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenTrie

fn default() -> Self

Returns the “default value” for a type.