Enum boa_parser::lexer::token::TokenKind
source · pub enum TokenKind {
Show 14 variants
BooleanLiteral((bool, ContainsEscapeSequence)),
EOF,
IdentifierName((Sym, ContainsEscapeSequence)),
PrivateIdentifier(Sym),
Keyword((Keyword, bool)),
NullLiteral(ContainsEscapeSequence),
NumericLiteral(Numeric),
Punctuator(Punctuator),
StringLiteral((Sym, EscapeSequence)),
TemplateNoSubstitution(TemplateString),
TemplateMiddle(TemplateString),
RegularExpressionLiteral(Sym, Sym),
LineTerminator,
Comment,
}
Expand description
Represents the type of Token and the data it has inside.
Variants§
BooleanLiteral((bool, ContainsEscapeSequence))
A boolean literal, which is either true
or false
.
EOF
The end of the file.
IdentifierName((Sym, ContainsEscapeSequence))
An identifier name.
PrivateIdentifier(Sym)
Keyword((Keyword, bool))
A keyword and a flag if the keyword contains unicode escaped chars.
For more information, see Keyword
.
NullLiteral(ContainsEscapeSequence)
The null
literal.
NumericLiteral(Numeric)
A numeric literal.
Punctuator(Punctuator)
A piece of punctuation
StringLiteral((Sym, EscapeSequence))
TemplateNoSubstitution(TemplateString)
A part of a template literal without substitution.
TemplateMiddle(TemplateString)
The part of a template literal between substitutions
RegularExpressionLiteral(Sym, Sym)
A regular expression, consisting of body and flags.
LineTerminator
Indicates a line terminator (\n
).
Comment
Indicates a comment, the content isn’t stored.
Implementations§
source§impl TokenKind
impl TokenKind
sourcepub const fn boolean_literal(lit: bool) -> Self
pub const fn boolean_literal(lit: bool) -> Self
Creates a BooleanLiteral
token kind.
sourcepub const fn identifier(ident: Sym) -> Self
pub const fn identifier(ident: Sym) -> Self
Creates an Identifier
token type.
sourcepub fn numeric_literal<L>(lit: L) -> Self
pub fn numeric_literal<L>(lit: L) -> Self
Creates a NumericLiteral
token kind.
sourcepub const fn punctuator(punc: Punctuator) -> Self
pub const fn punctuator(punc: Punctuator) -> Self
Creates a Punctuator
token type.
sourcepub const fn string_literal(lit: Sym, escape_sequence: EscapeSequence) -> Self
pub const fn string_literal(lit: Sym, escape_sequence: EscapeSequence) -> Self
Creates a StringLiteral
token type.
sourcepub const fn template_middle(template_string: TemplateString) -> Self
pub const fn template_middle(template_string: TemplateString) -> Self
Creates a TemplateMiddle
token type.
sourcepub const fn template_no_substitution(template_string: TemplateString) -> Self
pub const fn template_no_substitution(template_string: TemplateString) -> Self
Creates a TemplateNoSubstitution
token type.
sourcepub const fn regular_expression_literal(body: Sym, flags: Sym) -> Self
pub const fn regular_expression_literal(body: Sym, flags: Sym) -> Self
Creates a RegularExpressionLiteral
token kind.
sourcepub const fn line_terminator() -> Self
pub const fn line_terminator() -> Self
Creates a LineTerminator
token kind.