Crate strizer

Structs

StreamTokenizer

StreamTokenizer<R> defines the structure for the tokenization of a given input that implements the BufRead trait. By using BufRead, the content of the text data is efficiently tokenized by reading it line-by-line.

StringTokenizer

StringTokenizer defines the structure for the tokenization of strings.

Token

Token describes the primitive that is returned while iterating through a Tokenizer. Tokens can be of different type or kind:

Enums

TokenKind

TokenKind defines the different types of Token available.

Type Aliases

Span

Span defines the absolute range of a Token inside the input data.