Struct SourceTokens 

pub struct SourceTokens { /* private fields */ }

Lexer output for a source string without running the full module parser.

Use SourceFile when you also need the AST and parse diagnostics.

Implementations

impl SourceTokens

pub fn lex(source: &str) -> Self

Lexes source and records trivia and lexer errors.

pub fn tokens(&self) -> &[Token]

Raw lexer tokens in source order.

pub fn trivia(&self) -> &[Trivia]

Whitespace and comment trivia between tokens.

pub fn lex_errors(&self) -> &[LexError]

Lexer diagnostics for malformed input.

pub fn laid_out_tokens(&self) -> &[Token]

Tokens after layout resolution (virtual braces and semicolons inserted).

Trait Implementations

impl Debug for SourceTokens

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.