Struct Lexer

pub struct Lexer<'a> { /* private fields */ }

Implementations

impl<'a> Lexer<'a>

pub fn new() -> Self

Creates a new Lexer with the default alphabet.

This is conceptually equivalent of doing

use logic_parser::lexing::Lexer;

let mut lexer = Lexer::with_alphabets(
    |c| c.is_alphanumeric() || c == '_',
    |c| c.is_alphabetic() || c == '_'
);

pub fn with_alphabets( alphabet: fn(char) -> bool, start_chars_alphabet: fn(char) -> bool, ) -> Self

This allows you to define a custom alphabet for the lexer.

use logic_parser::lexing::Lexer;
use logic_parser::parsing::{Parser, ASTNode};

let query = "(tag:pink || tag:anime) && (mime:image/* || mime:video/*)";
let mut lexer = Lexer::with_alphabets(
    |c| c.is_alphanumeric() || c == '_' || c == ':' || c == '*' || c == '/',
    |c| c.is_alphabetic(),
);

let tokens = lexer.tokenize(query).unwrap();

let mut parser = Parser::new(&tokens);
parser.parse().unwrap();

WARNING

Be aware of the following:

• Creating an alphabet such that the start_chars_alphabet contains alphabet plus some other characters is undefined behaviour. It will probably loop forever.

pub fn with_alphabet(alphabet: fn(char) -> bool) -> Self

Creates a lexer that uses the same alphabet for the start characters and the rest.

Equivalent to doing

use logic_parser::lexing::Lexer;

let custom_alphabet = |c: char| c.is_alphanumeric() || c == '_';

let mut lexer = Lexer::with_alphabets(custom_alphabet, custom_alphabet);
lexer.tokenize("_puppies_").unwrap();

pub fn tokenize(&mut self, src: &'a str) -> Result<Vec<Token>>

Trait Implementations

impl<'a> Default for Lexer<'a>

fn default() -> Self

Returns the “default value” for a type.