Struct Lexer

pub struct Lexer<K> { /* private fields */ }

A regex-based lexer.

use regex_lexer::{LexerBuilder, Token};
 
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
enum Tok {
    Ident,
    // ...
}

let lexer = LexerBuilder::new()
    .token(r"\p{XID_Start}\p{XID_Continue}*", Tok::Ident)
    .ignore(r"\s+") // skip whitespace
    // ...
    .build()?;

let tokens = lexer.tokens("these are some identifiers");

Implementations

impl<K> Lexer<K>

pub fn builder<'r>() -> LexerBuilder<'r, K>

Create a LexerBuilder. This is the same as LexerBuilder::new.

pub fn tokens<'l, 't>(&'l self, source: &'t str) -> Tokens<'l, 't, K>

Return an iterator over all matched tokens.

Trait Implementations

impl<K: Debug> Debug for Lexer<K>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.