Function lexical_scanner::lexer_as_str

pub fn lexer_as_str(text: &str) -> Vec<Token>

Constructs a vector of tokens. This straight forward lexical scanner is preset to support over 75 tokens. The list of tokens can be found at this sites github page. Usually, passing in a text string is used for debugging and testing. Example

pub use lexical_scanner::*;
pub use enums::*;
let text = "The number 5.0 is > 1;";
let token_list = lexical_scanner::lexer_as_str(text);