Module tokenizer 

Re-exports

pub use token::Token;

pub use token::TokenKind;

Modules

token

Functions

tokenize

Returns an iterator over the token kinds produced by the tokenizer.

tokenize_complete

Returns an owned array containing all of the tokens produced by the tokenizer. This allows us to backtrack in case of an error.