[−][src]Module sql_ast::tokenizer
SQL Tokenizer
The tokenizer (a.k.a. lexer) converts a string into a sequence of tokens.
The tokens then form the input for the parser, which outputs an Abstract Syntax Tree (AST).
Structs
| Tokenizer | SQL Tokenizer |
| TokenizerError | Tokenizer error |
| Word | A keyword (like SELECT) or an optionally quoted SQL identifier |
Enums
| Token | SQL Token enumeration |
| Whitespace |