Crate sqlite3_tokenizer

Tokenizes SQL strings as SQLite would.

This is intended for implementors of SQLite libraries.

Example

for token in Tokenizer::new("SELECT * FROM t") {
    println!("Token of kind {:?} is written {:?}", token.kind, token.text);
}

outputs

Token of kind Select is written "SELECT"
Token of kind Space is written " "
Token of kind Star is written "*"
Token of kind Space is written " "
Token of kind From is written "FROM"
Token of kind Space is written " "
Token of kind Id is written "t"

Structs

Token	A single parsed token, describing its type and the string it came from
Tokenizer	Iterates over tokens in a string

Enums

TokenKind

Describes the kind of token recognized