Module tokenizer

Module tokenizer 

Source

Structsยง

Tokenizer
A grinder that combines character bundles into lexical tokens. This is the last stage of lexical analysis.