Crate strizer
Version 0.1.0
See all strizer's items
StreamTokenizer<R> defines the structure for the tokenization of a given input that implements the BufRead trait. By using BufRead, the content of the text data is efficiently tokenized by reading it line-by-line.
StreamTokenizer<R>
BufRead
StringTokenizer defines the structure for the tokenization of strings.
StringTokenizer
Token describes the primitive that is returned while iterating through a Tokenizer. Tokens can be of different type or kind:
Token
TokenKind defines the different types of Token available.
TokenKind
Span defines the absolute range of a Token inside the input data.
Span