#[non_exhaustive]
pub enum TokenType {
Eof,
Identifier {
identifier: ShortString,
},
MultiLineComment {
blocks: usize,
comment: ShortString,
},
Number {
text: ShortString,
},
Shebang {
line: ShortString,
},
SingleLineComment {
comment: ShortString,
},
StringLiteral {
literal: ShortString,
multi_line: Option<usize>,
quote_type: StringLiteralQuoteType,
},
Symbol {
symbol: Symbol,
},
Whitespace {
characters: ShortString,
},
InterpolatedString {
literal: ShortString,
kind: InterpolatedStringKind,
},
}
Expand description
The type of tokens in parsed code
Variants (Non-exhaustive)§
This enum is marked as non-exhaustive
Eof
End of file, should always be the very last token
Identifier
Fields
identifier: ShortString
The identifier itself
An identifier, such as foo
MultiLineComment
Fields
blocks: usize
Number of equals signs, if any, for the multi line comment
For example, --[=[
would have a blocks
value of 1
comment: ShortString
The comment itself, ignoring opening and closing tags
A multi line comment in the format of --[[ comment ]]
Number
Fields
text: ShortString
The text representing the number, includes details such as 0x
A literal number, such as 3.3
Shebang
Fields
line: ShortString
The shebang line itself
A shebang line
SingleLineComment
Fields
comment: ShortString
The comment, ignoring initial --
A single line comment, such as -- comment
StringLiteral
Fields
literal: ShortString
The literal itself, ignoring quotation marks
multi_line: Option<usize>
Number of equals signs used for a multi line string, if it is one
For example, [=[string]=]
would have a multi_line
value of Some(1)
[[string]]
would have a multi_line
value of Some(0)
A string such as "string"
would have a multi_line
value of None
quote_type: StringLiteralQuoteType
The type of quotation mark used to make the string
A literal string, such as “Hello, world”
Symbol
A Symbol
, such as local
or +
Whitespace
Fields
characters: ShortString
Characters consisting of the whitespace
Whitespace, such as tabs or new lines
InterpolatedString
Fields
literal: ShortString
The literal itself, ignoring backticks
kind: InterpolatedStringKind
The kind of interpolated string. If it is the beginning, middle, end, or a standalone string.
roblox
only.Some form of interpolated string
Implementations§
source§impl TokenType
impl TokenType
sourcepub fn is_trivia(&self) -> bool
pub fn is_trivia(&self) -> bool
Returns whether a token can be practically ignored in most cases
Comments and whitespace will return true
, everything else will return false