Crate lexical_scanner

The lexical_scanner procceses the user’s input and converts to a vector of 75+ various tokens. Lexical_scanner works using rust Iterator trait. Using Peekable(), the library can safely view and speedily identify character patterns without using regex.

Modules

enums

The lexical_scanner procceses supports over 75+ various tokens. All major punctuation patterns are supported.

lexer

Functions

lexer

Converts a file content’s to a Vector of Tokens
Input:: path: &str
Return -> Vec
Typically this is the main method for generating tokens by passing in a file path to the document you want to perform a lexical scan on.
Example

lexer_as_str

Converts a string to tokens
Input:: text: &str
Return -> Vec
This is comonnly used for debugging and testing.
Example