Enum full_moon::tokenizer::TokenType [−][src]
#[non_exhaustive] pub enum TokenType { Eof, Identifier { identifier: ShortString, }, MultiLineComment { blocks: usize, comment: ShortString, }, Number { text: ShortString, }, Shebang { line: ShortString, }, SingleLineComment { comment: ShortString, }, StringLiteral { literal: ShortString, multi_line: Option<usize>, quote_type: StringLiteralQuoteType, }, Symbol { symbol: Symbol, }, Whitespace { characters: ShortString, }, }
Expand description
The type of tokens in parsed code
Variants (Non-exhaustive)
This enum is marked as non-exhaustive
End of file, should always be the very last token
An identifier, such as foo
Fields of Identifier
identifier: ShortString
The identifier itself
A multi line comment in the format of --[[ comment ]]
Fields of MultiLineComment
blocks: usize
Number of equals signs, if any, for the multi line comment
For example, --[=[
would have a blocks
value of 1
comment: ShortString
The comment itself, ignoring opening and closing tags
A literal number, such as 3.3
Fields of Number
text: ShortString
The text representing the number, includes details such as 0x
A shebang line
Fields of Shebang
line: ShortString
The shebang line itself
A single line comment, such as -- comment
Fields of SingleLineComment
comment: ShortString
The comment, ignoring initial --
A literal string, such as “Hello, world”
Fields of StringLiteral
literal: ShortString
The literal itself, ignoring quotation marks
multi_line: Option<usize>
Number of equals signs used for a multi line string, if it is one
For example, [=[string]=]
would have a multi_line
value of Some(1)
[[string]]
would have a multi_line
value of Some(0)
A string such as "string"
would have a multi_line
value of None
quote_type: StringLiteralQuoteType
The type of quotation mark used to make the string
A Symbol
, such as local
or +
Fields of Symbol
symbol: Symbol
The symbol itself
Whitespace, such as tabs or new lines
Fields of Whitespace
characters: ShortString
Characters consisting of the whitespace
Implementations
Returns whether a token can be practically ignored in most cases
Comments and whitespace will return true
, everything else will return false
Returns the kind of the token type.
use full_moon::{ShortString, tokenizer::{TokenKind, TokenType}}; assert_eq!( TokenType::Identifier { identifier: ShortString::new("hello") }.kind(), TokenKind::Identifier, );
Trait Implementations
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
Auto Trait Implementations
impl RefUnwindSafe for TokenType
impl UnwindSafe for TokenType
Blanket Implementations
Mutably borrows from an owned value. Read more