Module sql_ast::tokenizer

SQL Tokenizer

The tokenizer (a.k.a. lexer) converts a string into a sequence of tokens.

The tokens then form the input for the parser, which outputs an Abstract Syntax Tree (AST).

Structs

• Tokenizer
  SQL Tokenizer
• TokenizerError
  Tokenizer error
• Word
  A keyword (like SELECT) or an optionally quoted SQL identifier

Enums

• Token
  SQL Token enumeration
• Whitespace