Struct Tokenizer

pub struct Tokenizer<'a> { /* private fields */ }

Tokenizer state on the data

Implementations

impl<'a> Tokenizer<'a>

pub fn new(data: &'a str) -> Self

Create a new tokenizer from the data stream

pub fn new_with_config(data: &'a str, cfg: TokenizerConfig) -> Self

Create a new tokenizer from the data stream with an associated config

pub fn next(&mut self) -> Result<Option<SpannedToken<'a>>, TokenError>

Return the next token, or none if reach the end of stream