Struct TokenStream 

pub struct TokenStream(/* private fields */);

An owned sequence of raw M2 token trees.

Implementations

impl TokenStream

pub fn new() -> Self

Creates an empty token stream.

pub fn iter( &self, ) -> impl DoubleEndedIterator<Item = &TokenTree> + ExactSizeIterator

Iterates over the stream’s top-level token trees.

Examples found in repository

examples/live_parser.rs (line 65)

fn render(source: &str, clear: bool) -> String {
    let mut output = String::new();
    if clear {
        output.push_str("\u{1b}[2J\u{1b}[H");
    }
    output.push_str("m2-syn live parser  (Ctrl-C/Ctrl-D exit, Backspace edits)\n\n");
    output.push_str("SOURCE\n");
    output.push_str(source);
    if !source.ends_with('\n') {
        output.push('\n');
    }

    let recovered_tokens = match lex_str_recovering(source) {
        Ok(tokens) => tokens,
        Err(error) => {
            output.push_str(&format!("\nSTATE\nlexical error: {error}\n"));
            return output;
        }
    };
    let cell_count = recovered_tokens
        .value
        .iter()
        .filter(|token| matches!(token, TokenTree::Eoc(_)))
        .count();
    output.push_str(&format!(
        "\nSTATE\n{} lexer cell(s); {} recoverable diagnostic(s)\n",
        cell_count,
        recovered_tokens.diagnostics.len()
    ));
    for diagnostic in &recovered_tokens.diagnostics {
        write_diagnostic(&mut output, diagnostic);
    }

    output.push_str("\nTOKENS\n");
    output.push_str(&format_pretty_tokens(&recovered_tokens.value));

    output.push_str("\nSYNTAX\n");
    match parse_native_recovering(source) {
        Ok(parsed) => {
            if parsed.diagnostics.is_empty() {
                output.push_str("complete\n");
            } else {
                output.push_str("incomplete but structurally recovered\n");
            }
            output.push_str(&format_pretty_tree(&parsed.value));
        }
        Err(error) => {
            output.push_str(&format!("expected parser state: {error}\n"));
        }
    }
    output
}

pub fn is_empty(&self) -> bool

Reports whether the stream contains no token trees.

pub fn pop(&mut self) -> Option<TokenTree>

Removes and returns the final token tree, if any.

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenStream

fn default() -> TokenStream

Returns the “default value” for a type.

impl Display for TokenStream

fn fmt(&self, formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenStream

impl<T: Lex> Extend<T> for TokenStream

Appends anything that can emit tokens, one item at a time.

Lex already states which types produce tokens; this states how a TokenStream accepts them, by forwarding to Lex::to_tokens. It subsumes the two concrete impls this replaced (TokenTree and TokenStream itself both implement Lex), and works for any other Lex type without a bespoke impl.

fn extend<I: IntoIterator<Item = T>>(&mut self, iter: I)

Extends a collection with the contents of an iterator.

fn extend_one(&mut self, item: A)

🔬This is a nightly-only experimental API. (extend_one)

Extends a collection with exactly one element.

fn extend_reserve(&mut self, additional: usize)

🔬This is a nightly-only experimental API. (extend_one)

Reserves capacity in a collection for the given number of additional elements.

impl FromIterator<TokenTree> for TokenStream

fn from_iter<T: IntoIterator<Item = TokenTree>>(iter: T) -> Self

Creates a value from an iterator.

impl IntoIterator for TokenStream

type Item = TokenTree

The type of the elements being iterated over.

type IntoIter = IntoIter<TokenTree>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl Lex for TokenStream

fn to_tokens(&self, output: &mut TokenStream)

Appends this value’s token trees to output.

fn to_token_stream(&self) -> TokenStream

Emits this value into a fresh token stream.

fn to_m2(&self) -> String

Renders this value as normalized Macaulay2 source.

impl Parse for TokenStream

fn parse(input: &mut ParseStream) -> Result<Self, TokenParseError>

Consumes this value from input.

impl PartialEq for TokenStream

fn eq(&self, other: &TokenStream) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Spanned for TokenStream

fn span(&self) -> Span

Returns the smallest span covering this value.

impl StructuralPartialEq for TokenStream