Enum full_moon::tokenizer::TokenType

pub enum TokenType<'a> {
    Eof,
    Identifier {
        identifier: Cow<'a, str>,
    },
    MultiLineComment {
        blocks: usize,
        comment: Cow<'a, str>,
    },
    Number {
        text: Cow<'a, str>,
    },
    SingleLineComment {
        comment: Cow<'a, str>,
    },
    StringLiteral {
        literal: Cow<'a, str>,
        multi_line: Option<usize>,
        quote_type: StringLiteralQuoteType,
    },
    Symbol {
        symbol: Symbol,
    },
    Whitespace {
        characters: Cow<'a, str>,
    },
}

The type of tokens in parsed code

Variants

Eof

End of file, should always be the very last token

Identifier

An identifier, such as foo

Fields of Identifier

identifier: Cow<'a, str>

The identifier itself

MultiLineComment

A multi line comment in the format of --[[ comment ]]

Fields of MultiLineComment

blocks: usize

Number of equals signs, if any, for the multi line comment For example, --[=[ would have a blocks value of 1

comment: Cow<'a, str>

The comment itself, ignoring opening and closing tags

Number

A literal number, such as 3.3

Fields of Number

text: Cow<'a, str>

The text representing the number, includes details such as 0x

SingleLineComment

A single line comment, such as -- comment

Fields of SingleLineComment

comment: Cow<'a, str>

The comment, ignoring initial --

StringLiteral

A literal string, such as "Hello, world"

Fields of StringLiteral

literal: Cow<'a, str>

The literal itself, ignoring quotation marks

multi_line: Option<usize>

Number of equals signs used for a multi line string, if it is one For example, [=[string]=] would have a multi_line value of Some(1) [[string]] would have a multi_line value of Some(0) A string such as "string" would have a multi_line value of None

quote_type: StringLiteralQuoteType

The type of quotation mark used to make the string

Symbol

A Symbol, such as local or +

Fields of Symbol

symbol: Symbol

The symbol itself

Whitespace

Whitespace, such as tabs or new lines

Fields of Whitespace

characters: Cow<'a, str>

Characters consisting of the whitespace

Implementations

impl<'a> TokenType<'a>

pub fn ignore(&self) -> bool

👎 Deprecated since 0.5.0:

Please use is_trivia instead

Returns whether a token can be practically ignored in most cases Comments and whitespace will return true, everything else will return false Deprecated in favor of TokenType::is_trivia, a name consistent with leading_trivia and trailing_trivia.

pub fn is_trivia(&self) -> bool

Returns whether a token can be practically ignored in most cases Comments and whitespace will return true, everything else will return false

pub fn kind(&self) -> TokenKind

Returns the TokenKind of the token type.

use std::borrow::Cow;
use full_moon::tokenizer::{TokenKind, TokenType};

assert_eq!(
    TokenType::Identifier {
        identifier: Cow::from("hello")
    }.kind(),
    TokenKind::Identifier,
);

pub fn spaces(spaces: usize) -> Self

Returns a whitespace TokenType consisting of spaces

pub fn tabs(tabs: usize) -> Self

Returns a whitespace TokenType consisting of tabs

Trait Implementations

impl<'a> Clone for TokenType<'a>

fn clone(&self) -> TokenType<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for TokenType<'a>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl<'de: 'a, 'a> Deserialize<'de> for TokenType<'a>

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
    __D: Deserializer<'de>, 

Deserialize this value from the given Serde deserializer.

impl<'a> Eq for TokenType<'a>

impl<'_> Owned for TokenType<'_>

type Owned = TokenType<'static>

What an owned version of the object looks like. Usually contains a 'static lifetime.

fn owned(&self) -> Self::Owned

Returns an owned version of the object.

impl<'a> PartialEq<TokenType<'a>> for TokenType<'a>

fn eq(&self, other: &TokenType<'a>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &TokenType<'a>) -> bool

This method tests for !=.

impl<'a> Serialize for TokenType<'a>

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error> where
    __S: Serializer, 

Serialize this value into the given Serde serializer.

impl<'a> StructuralEq for TokenType<'a>

impl<'a> StructuralPartialEq for TokenType<'a>