Module tokenizer

Module tokenizer 

Source
Expand description

Tokenizer holds a simple tokenizer that can tokenize a string into a vector of tokens

Enums§

Token
Token contains all the possible tokens that can be parsed

Functions§

tokenize
Tokenize the input string into a vector of tokens