Struct Lexer

pub struct Lexer<'a> { /* private fields */ }

The tokenizer you write will be passed a Lexer as an argument, and will call methods on it to figure out what the next lexeme is and, if needed, get its text. The Lexer keeps track of the column range of the lexeme.

Implementations

impl<'a> Lexer<'a>

pub fn next(&mut self) -> Option<char>

Returns the next character and adds it to the lexeme.

pub fn peek(&self) -> Option<char>

Returns the next character without advancing.

pub fn at(&mut self, c: char) -> bool

If the next character is c (the argument, not the letter), adds c to the lexeme and returns true. Otherwise just returns false.

pub fn take_while(&mut self, f: impl Fn(char) -> bool) -> &mut Self

Keeps adding characters to the lexeme while f(char) is true. Returns self to allow chaining.

pub fn skip_while(&mut self, f: impl Fn(char) -> bool) -> &mut Self

Skips over characters while f(char) is true; the lexeme will start at the char where f(char) is false. Returns self to allow chaining.

pub fn get(&self) -> &'a str

Returns the lexeme as an &str.

pub fn into<T: From<&'a str>>(&mut self) -> T

Returns the lexeme converted to the needed type (commonly a String). This is just a shortcut for .lexeme().into().

pub fn map<T>(&mut self, f: impl Fn(&'a str) -> T) -> T

Returns the result of applying f() to the lexeme.

pub fn column_range(&self) -> Range<usize>

Gets the range of columns associated with the lexeme. You probably won’t need to use this, since the column range is included in the Iterator’s output, but it could be useful if the interpretation of lexemes depends upon the columns in which you find them.

pub fn str_range(&self) -> Range<usize>

Gets the byte range of the line associated with the lexeme. You probably won’t need to use this, since .get() gives you the &str for the lexeme, but it could be useful if the interpretation of lexemes depends on the surrounding text.

pub fn line(&self) -> &str

Returns the line we are currently tokenizing. You probably won’t need to use this, but it could be useful if the interpretation of lexemes depends on the surrounding text.