Struct TokenStream 

pub struct TokenStream {
    pub spans: Vec<TokenSpan>,
    /* private fields */
}

Source-backed token stream (span-only).

This is intended as a high-performance representation for callers that want to avoid allocating a String per token. It can materialize owned Tokens when needed (FFI/debug).

Fields

spans: Vec<TokenSpan>

Implementations

impl TokenStream

pub fn new(formula: &str) -> Result<Self, TokenizerError>

pub fn new_with_dialect( formula: &str, dialect: FormulaDialect, ) -> Result<Self, TokenizerError>

pub fn new_best_effort(formula: &str) -> Self

pub fn new_best_effort_with_dialect( formula: &str, dialect: FormulaDialect, ) -> Self

pub fn diagnostics(&self) -> Vec<TokenDiagnostic>

pub fn diagnostics_ref(&self) -> &[TokenDiagnostic]

pub fn has_errors(&self) -> bool

pub fn invalid_spans_iter(&self) -> impl Iterator<Item = &TokenSpan>

pub fn invalid_spans(&self) -> Vec<&TokenSpan>

pub fn source(&self) -> &str

pub fn dialect(&self) -> FormulaDialect

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

pub fn get(&self, index: usize) -> Option<TokenView<'_>>

pub fn to_tokens(&self) -> Vec<Token>

pub fn render(&self) -> String

Reconstruct the tokenized payload from spans.

For formulas that start with ‘=’, this intentionally omits the leading ‘=’ to preserve historical TokenStream::render() behavior.

pub fn render_formula(&self) -> String

Reconstruct the full input formula, including a leading ‘=’ when present.

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.