Module polyglot_tokenizer::tokenizer

Structs

Tokenizer	The tokenizer is responsible for turning content into an iterator of Token.
Tokens

Enums

Token

Token is an enum whose variants represent each type of possible Token returned from the Tokenizer. Block Comments and Strings hold both the start and end indicator for the Tokens. Line Comments hold the open indicator for the Tokens. See below for examples.