Struct passerine::compiler::lex::Lexer

pub struct Lexer { /* fields omitted */ }

This represents a lexer object. A lexer takes a source file and lexes it into tokens. Note that this struct should not be controlled manually, use the lex function instead.

Implementations

impl Lexer

pub fn new(source: &Rc<Source>) -> Lexer

Create a new empty lexer.

pub fn all(&mut self) -> Result<Vec<Spanned<Token>>, Syntax>

Run the lexer, generating the entire token stream.

pub fn step(&self) -> Result<(Token, usize), String>

Step the lexer, returning the next token.

pub fn remaining(&self) -> &str

Helper function that returns the remaining source to be lexed as a &str.

pub fn strip(&mut self)

Helper function that Strips leading whitespace. Note that a newline is not leading whitespace, it’s a separator token.

pub fn expect(source: &str, literal: &str) -> Result<usize, String>

Helper function that expects an exact literal.

pub fn eat_digits(source: &str) -> Result<usize, String>

Helper function that eats numeric digits, returning how many lead.

pub fn literal(
    source: &str,
    literal: &str,
    kind: Token
) -> Result<(Token, usize), String>

Helper function that expects a literal, returning an error otherwise.

pub fn static_token(source: &str) -> Result<(Token, usize), String>

Parses a static token, like an operator or a keyword.

pub fn comment(source: &str) -> usize

Parses a single-line comment, which ignores from “–” until the next newline.

pub fn multi_comment(source: &str) -> usize

Parses a nestable multi-line comment, Which begins with -{ and ends with }-.

pub fn identifier(source: &str) -> Result<(Token, usize), String>

Classifies a symbol or a label. A series of alphanumerics and certain ascii punctuation (see Lexer::is_alpha). Can not start with a numeric character.

pub fn symbol(source: &str) -> Result<(Token, usize), String>

Classifies a symbol (i.e. variable name).

pub fn label(source: &str) -> Result<(Token, usize), String>

Classifies a label (i.e. data wrapper). Must start with an uppercase character.

pub fn keyword(source: &str) -> Result<(Token, usize), String>

Classifies a pseudokeyword, used in syntax macros. Must start with a single quote '.

pub fn real(source: &str) -> Result<(Token, usize), String>

Matches a number with a decimal point.

pub fn integer(source: &str) -> Result<(Token, usize), String>

pub fn string(source: &str) -> Result<(Token, usize), String>

Matches a string, converting escapes.

pub fn boolean(source: &str) -> Result<(Token, usize), String>

Matches a literal boolean.

pub fn sep(source: &str) -> Result<(Token, usize), String>

Matches a separator. Note that separators are special, as they’re mostly ignored They’re used to denote lines in functions blocks. A separator is either a newline or semicolon. They’re grouped, so something like ‘;\n’ is only one separator. Although the parser makes no assumptions, there should be only at most one separator between any two non-separator tokens.