Enum polyglot_tokenizer::tokenizer::Token

pub enum Token<'a> {
    BlockComment(&'a str, &'a str, &'a str),
    Ident(&'a str),
    LineComment(&'a str, &'a str),
    Number(&'a str),
    String(&'a str, &'a str, &'a str),
    Symbol(&'a str),
}

Token is an enum whose variants represent each type of possible Token returned from the Tokenizer. Block Comments and Strings hold both the start and end indicator for the Tokens. Line Comments hold the open indicator for the Tokens. See below for examples.

Examples

use polyglot_tokenizer::{Token, Tokenizer};

let content = "/* Block Comment */";
let tokens: Vec<Token> = Tokenizer::new(content).tokens().collect();
let expected = vec![Token::BlockComment("/*", " Block Comment ", "*/")];

assert_eq!(tokens, expected);

use polyglot_tokenizer::{Token, Tokenizer};

let content = "// Line Comment";
let tokens: Vec<Token> = Tokenizer::new(content).tokens().collect();
let expected = vec![Token::LineComment("//", "Line Comment")];

assert_eq!(tokens, expected);

Variants

BlockComment(&'a str, &'a str, &'a str)

Ident(&'a str)

LineComment(&'a str, &'a str)

Number(&'a str)

String(&'a str, &'a str, &'a str)

Symbol(&'a str)

Trait Implementations

impl<'a> Debug for Token<'a>

fn fmt(&self, f: &mut Formatter) -> Result

Formats the value using the given formatter.

impl<'a> Eq for Token<'a>

impl<'a> PartialEq<Token<'a>> for Token<'a>

fn eq(&self, other: &Token<'a>) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Token<'a>) -> bool

This method tests for !=.

impl<'a> StructuralEq for Token<'a>

impl<'a> StructuralPartialEq for Token<'a>