Struct LineToken 

pub struct LineToken {
    pub source_tokens: Vec<Token>,
    pub token_spans: Vec<Range<usize>>,
    pub line_type: LineType,
}

A line token represents one logical line created from grouped raw tokens.

Line tokens are produced by the line token transformation, which groups raw tokens into semantic line units. Each line token stores:

• The original raw tokens that created it (for location information and AST construction)
• The line type (what kind of line this is)
• Individual token spans (to enable byte-accurate text extraction from token subsets)

By preserving raw tokens and their individual spans, we can later pass them directly to existing AST constructors (using the same unified approach as the the parser), which handles all location tracking and AST node creation automatically.

Note: LineToken does NOT store an aggregate source_span. The AST construction facade will compute bounding boxes from the individual token_spans when needed.

Fields

source_tokens: Vec<Token>

The original raw tokens that comprise this line

token_spans: Vec<Range<usize>>

The byte range in source code for each token Must be the same length as source_tokens

line_type: LineType

The type/classification of this line

Implementations

impl LineToken

pub fn source_token_pairs(&self) -> Vec<(Token, Range<usize>)>

Get source tokens as (Token, Range) pairs.

This creates owned pairs from the separate source_tokens and token_spans vectors. Used by the AST construction facade to get tokens in the format expected by the token processing utilities.

Note: LineToken stores tokens and spans separately for serialization efficiency. This method creates the paired format needed for location tracking.

Trait Implementations

impl Clone for LineToken

fn clone(&self) -> LineToken

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LineToken

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for LineToken

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for LineToken

fn eq(&self, other: &LineToken) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for LineToken

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for LineToken