pub struct Token<'a> {
pub ty: TokenType,
pub literal: Cow<'a, str>,
pub line: usize,
pub col: usize,
pub detail: Option<&'static str>,
}Expand description
A lexical token with its type, literal value, and source position.
The literal text borrows from the lexer’s input ('a): identifiers,
keywords, numbers, parameters and escape-free string literals are slices
of the query text, and punctuation/operators and fixed error texts are
&'static str (which coerces to 'a). Only synthesized text is owned —
escape-decoded string literals and format!-built error messages — so
the token stream costs no heap allocation per token beyond the
Vec<Token> itself (ADR-0505). Tokens never outlive the input: the only
consumer is the [crate::parser::Parser], built and dropped inside the
parse entry points.
Fields§
§ty: TokenType§literal: Cow<'a, str>§line: usize§col: usize§detail: Option<&'static str>Optional symbolic openCypher-TCK error-detail name carried by a
TokenType::Error token (e.g. "InvalidNumberLiteral",
"InvalidUnicodeLiteral"). The parser propagates it onto the
crate::ParseError it raises when it surfaces the error token, so a
malformed number/unicode literal reports its specific TCK detail rather
than the generic UnexpectedSyntax. None for every non-error token.
Implementations§
Source§impl<'a> Token<'a>
impl<'a> Token<'a>
Sourcepub fn new(
ty: TokenType,
literal: impl Into<Cow<'a, str>>,
line: usize,
col: usize,
) -> Self
pub fn new( ty: TokenType, literal: impl Into<Cow<'a, str>>, line: usize, col: usize, ) -> Self
Build a token whose text is borrowed (&'a str / &'static str) or
owned (String), per the ADR-0505 literal rule — Into<Cow> accepts
all three without a hidden copy.
Sourcepub fn error_with_detail(
literal: impl Into<Cow<'a, str>>,
line: usize,
col: usize,
detail: &'static str,
) -> Self
pub fn error_with_detail( literal: impl Into<Cow<'a, str>>, line: usize, col: usize, detail: &'static str, ) -> Self
Build a TokenType::Error token tagged with a TCK error-detail name
(see Token::detail).