Struct SourceTokens 

pub struct SourceTokens { /* private fields */ }

Lexer output for a source string without running the full module parser.

Use SourceFile when you also need the AST and parse diagnostics.

Implementations

impl SourceTokens

pub fn lex(source: &str) -> Self

Lexes source and records trivia and lexer errors.

pub fn tokens(&self) -> &[Token]

Raw lexer tokens in source order.

pub fn trivia(&self) -> &[Trivia]

Whitespace and comment trivia between tokens.

pub fn lex_errors(&self) -> &[LexError]

Lexer diagnostics for malformed input.

pub fn laid_out_tokens(&self) -> &[Token]

Tokens after layout resolution (virtual braces and semicolons inserted).

Trait Implementations

impl Clone for SourceTokens

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SourceTokens

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for SourceTokens

impl PartialEq for SourceTokens

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.