Crate strizer

Structs

StreamTokenizer	StreamTokenizer<R> defines the structure for the tokenization of a given input that implements the BufRead trait. By using BufRead, the content of the text data is efficiently tokenized by reading it line-by-line.
StringTokenizer	StringTokenizer defines the structure for the tokenization of strings.
Token	Token describes the primitive that is returned while iterating through a Tokenizer. Tokens can be of different type or kind:

Enums

TokenKind

TokenKind defines the different types of Token available.

Type Definitions

Span	Span defines the absolute range of a Token inside the input data.