Module moore_svlog::cat

The categorizing lexer. Tokenizes an input stream of characters, yielding a stream of newline, whitespace, comment, symbol, and text tokens.

Example

extern crate moore_svlog_syntax;
let input = "Löwe 老虎 Léopard\n";
let mut cat = moore_svlog_syntax::cat::Cat::new(Box::new(input.char_indices()));
let tokens: Vec<_> = cat.collect();

Structs

Cat	The categorizing lexer. Divides an input stream of characters (unicode) into coarse groups of tokens. These include whitespace, comments, symbols, and text. The strings contained in the emitted tokens can be concatenated to arrive at the original file, i.e. no information is lost.
CatToken	A token emitted by the categorizing lexer.

Enums

CatTokenKind

The different kinds of tokens the categorizing lexer can emit.