Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Configurable tokenizer with custom operators, keywords, and middleware.

Built-in operators and keywords are always available. Custom registrations take priority over built-ins, checked longest-pattern-first.

let tok = Tokenizer::new()
    .op("=~", Token::Custom("RegexMatch".into()))
    .keyword("where", Token::Custom("Where".into()))
    .transform(|tokens| { /* rewrite and return */ tokens });
let tokens = tok.tokenize("name =~ 'foo.*'")?;

Implementations

impl Tokenizer

pub fn new() -> Self

Create a new tokenizer with only built-in operators and keywords.

pub fn op(self, pattern: &str, token: Token) -> Self

Register a custom operator pattern.

The pattern is matched byte-for-byte against the input. Custom ops are checked before built-in ops, longest pattern first.

pub fn keyword(self, word: &str, token: Token) -> Self

Register a custom keyword.

Keywords are matched case-insensitively against identifier tokens. Custom keywords override built-in keywords with the same name.

pub fn transform(self, f: fn(Vec<Token>) -> Vec<Token>) -> Self

Register a transform (middleware) that runs after tokenization.

Transforms run in registration order. Each receives the token stream and returns a new one. Use fn pointers for zero overhead.

pub fn tokenize(&self, input: &str) -> Result<Vec<Token>, String>

Tokenize input using this configuration.

Trait Implementations

impl Default for Tokenizer

fn default() -> Self

Returns the “default value” for a type.