Module molt::tokenizer

Tokenizer is a type used for parsing a &str into slices in a way not easily supported by the Peekable<Chars> iterator. The basic procedure is as follows:

• Use next and peek to query the iterator in the usual way.
• Detect the beginning of a token and get its index from mark.
• Skip just past the end of the token using next, skip, etc.
• Use token to retrieve a slice from the mark to the index.

Structs

Tokenizer

The Tokenizer type. See the module-level documentation.