Struct StackMaximaLexer 

pub struct StackMaximaLexer {
    pub decimal_sep: char,
    pub list_sep: char,
    pub end_tokens: Vec<char>,
    pub pm: bool,
    pub case_isensitive_keywords: bool,
    pub localised_keywords: HashMap<String, String>,
    pub lisp_ids: bool,
    /* private fields */
}

The actual lexer object.

Fields

decimal_sep: char

Decimal separator. Typically, '.'.

list_sep: char

List separator. Typically, ','.

end_tokens: Vec<char>

Supported end_tokens. Typically, vec![';','$'].

pm: bool

Do we support the #pm#-operator?

case_isensitive_keywords: bool

Are keywords case insensitive? Do we convert “tRuE” to “true”?

localised_keywords: HashMap<String, String>

Are there mappings for other keywords, in other languages?

lisp_ids: bool

Should we identify LISP-identifiers as tokens? Set to false if you work with questions, true if you work with the STACK-backend.

Implementations

impl StackMaximaLexer

pub fn new(input: String) -> StackMaximaLexer

pub fn set_source(&mut self, input: String)

Resets the token, position and char buffers of the lexer and set the input string.

pub fn next_token(&mut self) -> Option<StackMaximaToken>

Tries to get the next token. Either from the char-buffer or from a token buffer if any have been returned or previous actions have generated multiple tokens.

pub fn return_token(&mut self, token: StackMaximaToken)

Allows returning a token to the lexer, e.g., when the parser does its own insertions of virtual tokens.