Struct flexi_parse::TokenStream

pub struct TokenStream { /* private fields */ }

A sequence of tokens.

This is the return type of Group::token_stream, and can be created from a proc_macro::TokenStream or proc_macro2::TokenStream.

Implementations

impl TokenStream

pub fn prepare_whitespace(&mut self)

Removes all whitespace that doesn’t come at the start of a line.

Note that the parse* functions remove all whitespace.

pub fn remove_blank_space(&mut self)

Removes all non-newline whitespace from self.

Note that the parse* functions will remove all whitespace.

pub fn remove_whitespace(&mut self)

Removes all whitespace tokens from this stream.

This method is automatically called by the parse* functions.

pub fn is_empty(&self) -> bool

Returns true if there are no tokens in self.

Examples found in repository

examples/lox/main.rs (line 439)

    fn parse(input: ParseStream) -> Result<Self> {
        let name: Ident = input.parse()?;
        let mut contents: Group<Parentheses> = input.parse()?;
        contents.remove_whitespace();
        let Parentheses(span) = contents.delimiters();
        let tokens = contents.into_token_stream();
        let params: Vec<Ident> = if tokens.is_empty() {
            vec![]
        } else {
            let params: Punctuated<Ident, Punct![","]> =
                Punctuated::parse_separated.parse(tokens)?;
            params.into_iter().collect()
        };
        if params.len() >= 255 {
            input.add_error(input.new_error(
                "Can't have more than 254 parameters".to_string(),
                span,
                error_codes::TOO_MANY_ARGS,
            ));
        }
        let mut contents: Group<Braces> = input.parse()?;
        contents.remove_whitespace();
        let body = block.parse(contents.into_token_stream())?;
        Ok(Function { name, params, body })
    }

pub fn append(&mut self, other: &mut TokenStream)

Add another TokenStream to the end of self.

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<A: ToTokens> Extend<A> for TokenStream

fn extend<T: IntoIterator<Item = A>>(&mut self, iter: T)

Extends a collection with the contents of an iterator.

fn extend_one(&mut self, item: A)

🔬This is a nightly-only experimental API. (extend_one)

Extends a collection with exactly one element.

fn extend_reserve(&mut self, additional: usize)

🔬This is a nightly-only experimental API. (extend_one)

Reserves capacity in a collection for the given number of additional elements.

impl<'a> From<&'a TokenStream> for ParseBuffer<'a>

fn from(value: &'a TokenStream) -> Self

Converts to this type from the input type.

impl<'a> From<TokenStream> for ParseBuffer<'a>

fn from(value: TokenStream) -> Self

Converts to this type from the input type.

impl From<TokenStream> for TokenStream

Available on crate features proc-macro and proc-macro2 only.

fn from(value: TokenStream) -> Self

Converts to this type from the input type.

impl From<TokenStream> for TokenStream

Available on crate feature proc-macro2 only.

fn from(value: TokenStream2) -> Self

Converts to this type from the input type.

impl Hash for TokenStream

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for TokenStream

fn eq(&self, other: &TokenStream) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for TokenStream

fn partial_cmp(&self, other: &TokenStream) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

This method tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

This method tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

This method tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

This method tests greater than or equal to (for self and other) and is used by the >= operator.

impl ToTokens for TokenStream

fn to_tokens(&self, tokens: &mut TokenStream)

Append self to the given TokenStream.

fn to_token_stream(&self) -> TokenStream

Convert self directly into a TokenStream.

fn into_token_stream(self) -> TokenStream

where Self: Sized,

Convert self directly into a TokenStream.

impl ToTokens for TokenStream

Available on crate feature proc-macro2 only.

fn to_tokens(&self, tokens: &mut TokenStream2)

Write self to the given TokenStream.

fn to_token_stream(&self) -> TokenStream2

Convert self directly into a TokenStream object.

fn into_token_stream(self) -> TokenStream2

Convert self directly into a TokenStream object.

impl<'a> TryFrom<&'a TokenStream> for TokenStream

Available on crate feature proc-macro2 only.

type Error = Error

The type returned in the event of a conversion error.

fn try_from(value: &'a TokenStream) -> Result<Self, Self::Error>

Performs the conversion.

impl TryFrom<Arc<SourceFile>> for TokenStream

type Error = Error

The type returned in the event of a conversion error.

fn try_from(value: Arc<SourceFile>) -> Result<Self>

Performs the conversion.

impl TryFrom<TokenStream> for TokenStream

Available on crate feature proc-macro2 only.

type Error = Error

The type returned in the event of a conversion error.

fn try_from(value: TokenStream) -> Result<Self>

Performs the conversion.

impl StructuralPartialEq for TokenStream