Struct Tokenizer

pub struct Tokenizer { /* private fields */ }

Holds a stream of tokens.

Implementations

impl Tokenizer

pub fn from(input: String) -> Self

Constructs a new token stream from a String.

pub fn peek(&self) -> Option<Token>

Peeks at the next character in the stream.

pub fn next(&mut self) -> Option<Token>

Advances the character stream.

pub fn get_tokens(&mut self) -> Vec<Token>

Returns all tokens without consuming the tokenizer.

pub fn chk_silent(&self) -> bool

Checks whether or not the last token is a semicolon.

Lines that end with semicolons are not displayed.

pub fn get_next_precedence(&self) -> u8

Get the precedence of the next token.