Enum full_moon::tokenizer::TokenType

#[non_exhaustive]
pub enum TokenType<'a> {
    Eof,
    Identifier {
        identifier: Cow<'a, str>,
    },
    MultiLineComment {
        blocks: usize,
        comment: Cow<'a, str>,
    },
    Number {
        text: Cow<'a, str>,
    },
    Shebang {
        line: Cow<'a, str>,
    },
    SingleLineComment {
        comment: Cow<'a, str>,
    },
    StringLiteral {
        literal: Cow<'a, str>,
        multi_line: Option<usize>,
        quote_type: StringLiteralQuoteType,
    },
    Symbol {
        symbol: Symbol,
    },
    Whitespace {
        characters: Cow<'a, str>,
    },
}

The type of tokens in parsed code

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Eof

End of file, should always be the very last token

Identifier

An identifier, such as foo

Fields of Identifier

identifier: Cow<'a, str>

The identifier itself

MultiLineComment

A multi line comment in the format of --[[ comment ]]

Fields of MultiLineComment

blocks: usize

Number of equals signs, if any, for the multi line comment For example, --[=[ would have a blocks value of 1

comment: Cow<'a, str>

The comment itself, ignoring opening and closing tags

Number

A literal number, such as 3.3

Fields of Number

text: Cow<'a, str>

The text representing the number, includes details such as 0x

Shebang

A shebang line

Fields of Shebang

line: Cow<'a, str>

The shebang line itself

SingleLineComment

A single line comment, such as -- comment

Fields of SingleLineComment

comment: Cow<'a, str>

The comment, ignoring initial --

StringLiteral

A literal string, such as “Hello, world”

Fields of StringLiteral

literal: Cow<'a, str>

The literal itself, ignoring quotation marks

multi_line: Option<usize>

Number of equals signs used for a multi line string, if it is one For example, [=[string]=] would have a multi_line value of Some(1) [[string]] would have a multi_line value of Some(0) A string such as "string" would have a multi_line value of None

quote_type: StringLiteralQuoteType

The type of quotation mark used to make the string

Symbol

A Symbol, such as local or +

Fields of Symbol

symbol: Symbol

The symbol itself

Whitespace

Whitespace, such as tabs or new lines

Fields of Whitespace

characters: Cow<'a, str>

Characters consisting of the whitespace

Implementations

impl<'a> TokenType<'a>

pub fn is_trivia(&self) -> bool

Returns whether a token can be practically ignored in most cases Comments and whitespace will return true, everything else will return false

pub fn kind(&self) -> TokenKind

Returns the kind of the token type.

use std::borrow::Cow;
use full_moon::tokenizer::{TokenKind, TokenType};

assert_eq!(
    TokenType::Identifier {
        identifier: Cow::from("hello")
    }.kind(),
    TokenKind::Identifier,
);

pub fn spaces(spaces: usize) -> Self

Returns a whitespace TokenType consisting of spaces

pub fn tabs(tabs: usize) -> Self

Returns a whitespace TokenType consisting of tabs

Trait Implementations

impl<'a> Clone for TokenType<'a>

fn clone(&self) -> TokenType<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for TokenType<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de: 'a, 'a> Deserialize<'de> for TokenType<'a>

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl<'a> From<TokenType<'a>> for Result<TokenType<'a>, TokenizerErrorType>

fn from(token_type: TokenType<'a>) -> Result<TokenType<'a>, TokenizerErrorType>

Performs the conversion.

impl Owned for TokenType<'_>

type Owned = TokenType<'static>

What an owned version of the object looks like. Usually contains a 'static lifetime.

fn owned(&self) -> Self::Owned

Returns an owned version of the object.

impl<'a> PartialEq<TokenType<'a>> for TokenType<'a>

fn eq(&self, other: &TokenType<'a>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &TokenType<'a>) -> bool

This method tests for !=.

impl<'a> Serialize for TokenType<'a>

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.

impl<'a> Eq for TokenType<'a>

impl<'a> StructuralEq for TokenType<'a>

impl<'a> StructuralPartialEq for TokenType<'a>