Struct Token 

pub struct Token {
    pub kind: SyntaxKind,
    pub span: Span,
    pub preceded_by_newline: bool,
}

A token the lexer emits before it is folded into the lossless tree: its kind, the source span it covers, and whether a line break sits in front of it.

The parser consumes these into a rowan::GreenNode; the spans are kept so diagnostics can point at lexer-level locations (§6).

Fields

kind: SyntaxKind

span: Span

preceded_by_newline: bool

True iff the trivia run immediately before this token contained a \n or \r, or was a line comment.

A newline is trivia, so folding it into the tree loses the one fact statement separation needs: whether the next token starts a new line (D8, ADR-049). Recording it on the token is what lets the parser ask without re-reading the source, and what keeps the answer available after the trivia has already been emitted into the green tree.

Implementations

impl Token

pub fn new(kind: SyntaxKind, span: Span, preceded_by_newline: bool) -> Token

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Token

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Token

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token