Struct flexi_parse::TokenStream

pub struct TokenStream { /* private fields */ }

A sequence of tokens.

This is the return type of Group::token_stream, and can be created from a proc_macro::TokenStream or proc_macro2::TokenStream.

Implementations

impl TokenStream

pub fn prepare_whitespace(&mut self)

Removes all whitespace that doesn’t come at the start of a line.

Note that the parse* functions remove all whitespace.

pub fn remove_blank_space(&mut self)

Removes all non-newline whitespace from self.

Note that the parse* functions will remove all whitespace.

pub fn remove_whitespace(&mut self)

Removes all whitespace tokens from this stream.

This method is automatically called by the parse* functions.

pub fn is_empty(&self) -> bool

Returns true if there are no tokens in self.

Examples found in repository

examples/lox/main.rs (line 436)

    fn parse(input: ParseStream<'_>) -> Result<Self> {
        let name: Ident = input.parse()?;
        let mut contents: Group<Parentheses> = input.parse()?;
        contents.remove_whitespace();
        let Parentheses(span) = contents.delimiters();
        let tokens = contents.into_token_stream();
        let params: Vec<Ident> = if tokens.is_empty() {
            vec![]
        } else {
            let params: Punctuated<Ident, Punct![","]> =
                Punctuated::parse_separated.parse(tokens)?;
            params.into_iter().collect()
        };
        if params.len() >= 255 {
            input.add_error(input.new_error(
                "Can't have more than 254 parameters".to_string(),
                span,
                error_codes::TOO_MANY_ARGS,
            ));
        }
        let mut contents: Group<Braces> = input.parse()?;
        contents.remove_whitespace();
        let body = block.parse(contents.into_token_stream())?;
        Ok(Function { name, params, body })
    }

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<TokenStream> for TokenStream

Available on crate features proc-macro and proc-macro2 only.

fn from(value: TokenStream) -> Self

Converts to this type from the input type.

impl From<TokenStream> for TokenStream

Available on crate feature proc-macro2 only.

fn from(value: TokenStream2) -> Self

Converts to this type from the input type.

impl PartialEq<TokenStream> for TokenStream

fn eq(&self, other: &TokenStream) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl TryFrom<Rc<SourceFile, Global>> for TokenStream

type Error = Error

The type returned in the event of a conversion error.

fn try_from(value: Rc<SourceFile>) -> Result<Self>

Performs the conversion.

impl StructuralPartialEq for TokenStream