Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

A tokenizer for .tree files.

This struct represents the state of the tokenizer. It is not tied to any particular input, while TokenizerI is.

Implementations

impl Tokenizer

pub const fn new() -> Self

Create a new tokenizer.

pub fn tokenize(&mut self, text: &str) -> Result<Vec<Token>, Error>

Tokenize the input .tree text.

tokenize is the entry point of the Tokenizer. It takes a string of .tree text and returns a vector of tokens.

Trait Implementations

impl Default for Tokenizer

fn default() -> Self

Returns the “default value” for a type.