Struct TokenStream 

pub struct TokenStream {
    pub spans: Vec<TokenSpan>,
    /* private fields */
}

Source-backed token stream (span-only).

This is intended as a high-performance representation for callers that want to avoid allocating a String per token. It can materialize owned Tokens when needed (FFI/debug).

Fields

spans: Vec<TokenSpan>

Implementations

impl TokenStream

pub fn new(formula: &str) -> Result<Self, TokenizerError>

pub fn new_with_dialect( formula: &str, dialect: FormulaDialect, ) -> Result<Self, TokenizerError>

pub fn source(&self) -> &str

pub fn dialect(&self) -> FormulaDialect

pub fn len(&self) -> usize

pub fn is_empty(&self) -> bool

pub fn get(&self, index: usize) -> Option<TokenView<'_>>

pub fn to_tokens(&self) -> Vec<Token>

pub fn render(&self) -> String

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.