Struct passerine::compiler::lex::Lexer

pub struct Lexer { /* fields omitted */ }

This represents a lexer object. A lexer takes a source file and lexes it into tokens. Note that this struct should not be controlled manually, use the lex function instead.

Implementations

impl Lexer

pub fn new(source: &Rc<Source>) -> Lexer

Create a new empty lexer.

pub fn all(&mut self) -> Result<Vec<Spanned<Token>>, Syntax>

Run the lexer, generating the entire token stream.

pub fn step(&self) -> Result<(Token, usize), String>

Step the lexer, returning the next token.

pub fn remaining(&self) -> &str

Helper function that returns the remaining source to be lexed as a &str.

pub fn strip(&mut self)

Helper function that Strips leading whitespace. Note that a newline is not leading whitespace, it's a separator token.

pub fn expect(source: &str, literal: &str) -> Result<usize, String>

Helper function that expects an exact literal.

pub fn eat_digits(source: &str) -> Result<usize, String>

Helper function that eats numeric digits, returning how many lead.

pub fn literal(
    source: &str,
    literal: &str,
    kind: Token
) -> Result<(Token, usize), String>

Helper function that expects a literal, returning an error otherwise.

pub fn open_bracket(source: &str) -> Result<(Token, usize), String>

Matches a literal opening bracket {.

pub fn close_bracket(source: &str) -> Result<(Token, usize), String>

Matches a literal closing bracket `{``.

pub fn unit(source: &str) -> Result<(Token, usize), String>

Matches a literal closing parenthesis ).

pub fn open_paren(source: &str) -> Result<(Token, usize), String>

Matches a literal opening parenthesis (.

pub fn close_paren(source: &str) -> Result<(Token, usize), String>

Matches a literal closing parenthesis ).

pub fn assign(source: &str) -> Result<(Token, usize), String>

Matches a literal assignment equal sign =.

pub fn lambda(source: &str) -> Result<(Token, usize), String>

Matches a literal lambda arrow ->.

pub fn print(source: &str) -> Result<(Token, usize), String>

Matches a print expression.

pub fn symbol(source: &str) -> Result<(Token, usize), String>

Classifis a symbol (i.e. variable name). for now, a symbol is one or more ascii alphanumerics

pub fn real(source: &str) -> Result<(Token, usize), String>

Matches a number with a decimal point.

pub fn string(source: &str) -> Result<(Token, usize), String>

Matches a string, converting escapes.

pub fn boolean(source: &str) -> Result<(Token, usize), String>

Matches a literal boolean.

pub fn sep(source: &str) -> Result<(Token, usize), String>

Matches a separator. Note that separators are special, as they're mostly ignored They're used to denote lines in functions blocks. A separator is either a newline or semicolon. They're grouped, so something like ';\n' is only one separator. Although the parser makes no assumptions, there should be only at most one separator between any two non-separator tokens.