Enum full_moon::tokenizer::TokenType

#[non_exhaustive]
pub enum TokenType {
    Eof,
    Identifier {
        identifier: ShortString,
    },
    MultiLineComment {
        blocks: usize,
        comment: ShortString,
    },
    Number {
        text: ShortString,
    },
    Shebang {
        line: ShortString,
    },
    SingleLineComment {
        comment: ShortString,
    },
    StringLiteral {
        literal: ShortString,
        multi_line: Option<usize>,
        quote_type: StringLiteralQuoteType,
    },
    Symbol {
        symbol: Symbol,
    },
    Whitespace {
        characters: ShortString,
    },
}

The type of tokens in parsed code

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Eof

End of file, should always be the very last token

Identifier

Fields

identifier: ShortString

The identifier itself

An identifier, such as foo

MultiLineComment

Fields

blocks: usize

Number of equals signs, if any, for the multi line comment For example, --[=[ would have a blocks value of 1

comment: ShortString

The comment itself, ignoring opening and closing tags

A multi line comment in the format of --[[ comment ]]

Number

Fields

text: ShortString

The text representing the number, includes details such as 0x

A literal number, such as 3.3

Shebang

Fields

line: ShortString

The shebang line itself

A shebang line

SingleLineComment

Fields

comment: ShortString

The comment, ignoring initial --

A single line comment, such as -- comment

StringLiteral

Fields

literal: ShortString

The literal itself, ignoring quotation marks

multi_line: Option<usize>

Number of equals signs used for a multi line string, if it is one For example, [=[string]=] would have a multi_line value of Some(1) [[string]] would have a multi_line value of Some(0) A string such as "string" would have a multi_line value of None

quote_type: StringLiteralQuoteType

The type of quotation mark used to make the string

A literal string, such as “Hello, world”

Symbol

Fields

symbol: Symbol

The symbol itself

A Symbol, such as local or +

Whitespace

Fields

characters: ShortString

Characters consisting of the whitespace

Whitespace, such as tabs or new lines

Implementations

impl TokenType

pub fn is_trivia(&self) -> bool

Returns whether a token can be practically ignored in most cases Comments and whitespace will return true, everything else will return false

pub fn kind(&self) -> TokenKind

Returns the kind of the token type.

use full_moon::{ShortString, tokenizer::{TokenKind, TokenType}};

assert_eq!(
    TokenType::Identifier {
        identifier: ShortString::new("hello")
    }.kind(),
    TokenKind::Identifier,
);

pub fn spaces(spaces: usize) -> Self

Returns a whitespace TokenType consisting of spaces

pub fn tabs(tabs: usize) -> Self

Returns a whitespace TokenType consisting of tabs

Trait Implementations

impl Clone for TokenType

fn clone(&self) -> TokenType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenType

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl From<TokenType> for Result<TokenType, TokenizerErrorType>

fn from(token_type: TokenType) -> Result<TokenType, TokenizerErrorType>

Performs the conversion.

impl PartialEq<TokenType> for TokenType

fn eq(&self, other: &TokenType) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &TokenType) -> bool

This method tests for !=.

impl Serialize for TokenType

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.

impl Eq for TokenType

impl StructuralEq for TokenType

impl StructuralPartialEq for TokenType