Struct molt::tokenizer::Tokenizer

pub struct Tokenizer<'a> { /* fields omitted */ }

The Tokenizer type. See the module-level documentation.

Methods

impl<'a> Tokenizer<'a>

pub fn new(input: &'a str) -> Self

Creates a new tokenizer for the given input.

pub fn input(&self) -> &str

Returns the entire input.

pub fn as_str(&self) -> &str

pub fn mark(&self) -> usize

pub fn tail(&self, mark: usize) -> &str

pub fn next(&mut self) -> Option<char>

Returns the next character and updates the index.

pub fn peek(&mut self) -> Option<char>

Returns the next character without updating the index.

pub fn token(&self, mark: usize) -> &str

Get the token between the mark and the index. Returns "" if we're at the end or mark == index.

pub fn token2(&self, mark: usize, index: usize) -> &str

Get the token between the mark and the index. Returns "" if mark == index.

pub fn is(&mut self, ch: char) -> bool

Is the next character the given character? Does not update the index.

pub fn has<P>(&mut self, predicate: P) -> bool where
    P: Fn(&char) -> bool, 

Is the predicate true for the next character? Does not update the index.

pub fn at_end(&mut self) -> bool

Is there anything left in the input?

pub fn skip(&mut self)

Skip over the next character, updating the index. This is equivalent to next, but communicates better.

pub fn skip_char(&mut self, ch: char)

Skip over the given character, updating the index. This is equivalent to next, but communicates better. Panics if the character is not matched.

pub fn skip_over(&mut self, num_chars: usize)

Skips the given number of characters, updating the index. It is not an error if the iterator doesn't contain that many.

pub fn skip_while<P>(&mut self, predicate: P) where
    P: Fn(&char) -> bool, 

Skips over characters while the predicate is true. Updates the index.

pub fn backslash_subst(&mut self) -> char

Parses a backslash-escape and returns its value. If the escape is valid, the value will be the substituted character. If the escape is not valid, it will be the single character following the backslash. Either way, the the index will point at what's next. If there's nothing following the backslash, return the backslash.

Trait Implementations

impl<'a> Clone for Tokenizer<'a>

fn clone(&self) -> Tokenizer<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Tokenizer<'a>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.