Struct TokenStream 

pub struct TokenStream { /* private fields */ }

Represents a stream of tokens, typically derived from a log line.

A TokenStream stores a sequence of (Offset, Token) pairs, preserving the original position and type of each token within the source string.

Implementations

impl TokenStream

pub fn from_unicode_line(line: &str) -> Self

Creates a TokenStream from a Unicode log line.

This function splits the input line by ASCII whitespace and attempts to identify and intern each word as a Token::Value(TypedToken::String). It also calculates and stores the Offset for each token.

Arguments

• line - The input log line as a string slice.

Returns

A new TokenStream instance.

pub fn first(&self) -> Option<Token>

Returns the first Token in the stream.

Returns

An Option<Token> containing a clone of the first token if the stream is not empty, otherwise None.

pub fn len(&self) -> usize

Returns the number of tokens in the stream.

Returns

The length of the token stream as a usize.

pub fn is_empty(&self) -> bool

Checks if the token stream is empty.

Returns

true if the token stream contains no tokens, false otherwise.

pub fn get_token_at_index(&self, idx: usize) -> Option<Token>

Returns a clone of the Token at the specified index.

Arguments

• idx - The zero-based index of the token to retrieve.

Returns

An Option<Token> containing a clone of the token if the index is valid, otherwise None.

Trait Implementations

impl Clone for TokenStream

fn clone(&self) -> TokenStream

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenStream

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the TokenStream for display.

This implementation reconstructs the original string from the tokens and their offsets, preserving the original whitespace between tokens.

Arguments

• f - The formatter to write into.

Returns

A fmt::Result indicating success or failure of the formatting operation.

impl PartialEq for TokenStream

fn eq(&self, other: &TokenStream) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenStream