Struct TokenStream

pub struct TokenStream<'a> { /* private fields */ }

A stream of tokens. This is a wrapper around a logos::Lexer. It implements Clone, so it can be cloned and used to peek ahead. It also implements Iterator, so it can be used to iterate over the tokens.

Implementations

impl<'a> TokenStream<'a>

pub fn new(input: &'a str) -> Self

Creates a new token stream from the given input string. The input string is not copied, so it must outlive the token stream.

pub fn len(&self) -> usize

Returns the remaining input string.

pub fn count(&self) -> usize

Returns the number of remaining tokens in the stream.

pub fn is_empty(&self) -> bool

Returns true if the token stream is empty.

pub fn peek(&self) -> Option<(Result<Token, ParserError>, &'a str)>

Returns the next token in the stream, or None if the stream is empty.

pub fn peek_n( &self, n: usize, ) -> Option<Vec<(Result<Token, ParserError>, String)>>

Returns the next n tokens in the stream. If there are fewer than n tokens left, returns the remaining tokens. If the stream is empty, returns None.

pub fn advance(self) -> Self

Skips the next token in the stream.

pub fn span(&self) -> Range<usize>

Returns the span of the current token.

Trait Implementations

impl Clone for TokenStream<'_>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for TokenStream<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> From<&'a String> for TokenStream<'a>

fn from(s: &'a String) -> Self

Converts to this type from the input type.

impl<'a> From<&'a str> for TokenStream<'a>

fn from(s: &'a str) -> Self

Converts to this type from the input type.

impl InputLength for TokenStream<'_>

fn input_len(&self) -> usize

Calculates the input length, as indicated by its name, and the name of the trait itself

impl<'a> Parser<TokenStream<'a>, &'a str, ParserError> for Token

fn parse( &mut self, input: TokenStream<'a>, ) -> IResult<TokenStream<'a>, &'a str, ParserError>

A parser takes in input type, and returns a Result containing either the remaining input and the output value, or an error

fn map<G, O2>(self, g: G) -> Map<Self, G, O>

where G: Fn(O) -> O2, Self: Sized,

Maps a function over the result of a parser

fn flat_map<G, H, O2>(self, g: G) -> FlatMap<Self, G, O>

where G: FnMut(O) -> H, H: Parser<I, O2, E>, Self: Sized,

Creates a second parser from the output of the first one, then apply over the rest of the input

fn and_then<G, O2>(self, g: G) -> AndThen<Self, G, O>

where G: Parser<O, O2, E>, Self: Sized,

Applies a second parser over the output of the first one

fn and<G, O2>(self, g: G) -> And<Self, G>

where G: Parser<I, O2, E>, Self: Sized,

Applies a second parser after the first one, return their results as a tuple

fn or<G>(self, g: G) -> Or<Self, G>

where G: Parser<I, O, E>, Self: Sized,

Applies a second parser over the input if the first one failed

fn into<O2, E2>(self) -> Into<Self, O, O2, E, E2>

where O2: From<O>, E2: From<E>, Self: Sized,

automatically converts the parser’s output and error values to another type, as long as they implement the From trait

impl ToString for TokenStream<'_>

fn to_string(&self) -> String

Converts the given value to a String.