Struct Token 

pub struct Token { /* private fields */ }

Token describes the primitive that is returned while iterating through a Tokenizer. Tokens can be of different type or kind:

• TokenKind::Character
• TokenKind::Word.

The associated data is stored in the Token, along with its start position (in bytes).

Implementations

impl Token

pub fn kind(&self) -> &TokenKind

Returns token’s kind

pub fn character(character: char) -> Token

Creates a character token with a given start position (in bytes).

pub fn is_character(&self) -> bool

Returns true if the Token is TokenKind::Character

pub fn is_character_equal(&self, input: char) -> bool

Returns true if the Token is TokenKind::Character and equal to the input character.

pub fn word() -> Token

Creates a word token

pub fn is_word(&self) -> bool

Returns true if the Token is TokenKind::Word

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for Token

impl Eq for Token

impl StructuralPartialEq for Token