Struct TypedTokenizer 

pub struct TypedTokenizer<G: TypedGrammar> { /* private fields */ }

Tokenizer parameterized by grammar type G.

Useful for reusable tooling built against generated grammars.

• Use this when grammar type is known at compile time.
• Use Tokenizer for typical SQLite SQL app code.

Implementations

impl<G: TypedGrammar> TypedTokenizer<G>

pub fn new(grammar: G) -> Self

Create a tokenizer for grammar G.

Examples

use syntaqlite_syntax::typed::{grammar, TypedTokenizer};

let _tokenizer = TypedTokenizer::new(grammar());

Panics

Panics if tokenizer allocation fails (out of memory).

pub fn tokenize<'a>( &self, source: &'a str, ) -> impl Iterator<Item = TypedToken<'a, G>> + use<'a, G>

Tokenize source and iterate typed tokens.

The source is copied; the original does not need to outlive the iterator. For zero-copy tokenization use tokenize_cstr.

Examples

use syntaqlite_syntax::TokenType;
use syntaqlite_syntax::typed::{grammar, TypedTokenizer};

let tokenizer = TypedTokenizer::new(grammar());
let tokens: Vec<_> = tokenizer.tokenize("SELECT 1").collect();

assert_eq!(tokens[0].token_type(), TokenType::Select);
assert_eq!(tokens[0].text(), "SELECT");

Panics

Panics if another cursor from this tokenizer is still active. Drop the previous iterator before starting a new one.

pub fn tokenize_cstr<'a>( &self, source: &'a CStr, ) -> impl Iterator<Item = TypedToken<'a, G>> + use<'a, G>

Zero-copy tokenization over a null-terminated source buffer.

No copy is performed. The source must be valid UTF-8 (panics otherwise).

Examples

use std::ffi::CString;
use syntaqlite_syntax::TokenType;
use syntaqlite_syntax::typed::{grammar, TypedTokenizer};

let tokenizer = TypedTokenizer::new(grammar());
let sql = CString::new("SELECT 1").unwrap();
let types: Vec<_> = tokenizer.tokenize_cstr(&sql).map(|t| t.token_type()).collect();

assert!(types.contains(&TokenType::Select));

Panics

Panics if another cursor from this tokenizer is still active, or if source is not valid UTF-8.