Struct Token 

pub struct Token {
    pub kind: Kind,
    pub flags: Flags,
    pub keyword: u16,
    pub start: u32,
    pub end: u32,
}

One token. Twelve bytes, no allocation, no owned text.

Fields

kind: Kind

What it is.

flags: Flags

What was in the gap before it.

keyword: u16

The index into generated::keywords::KEYWORDS, or NOT_A_KEYWORD.

Resolved once here rather than at every position the matcher considers the token, because the matcher considers most tokens many times and the fold plus binary search is the expensive part of asking.

start: u32

Where it starts, as a byte offset into the query.

end: u32

One past where it ends.

Implementations

impl Token

pub const fn span(self) -> Span

The span this token covers.

pub fn text(self, query: &str) -> &str

The text of this token, given the query it came from.

Both bounds came from a scan over the same string and always land on a character boundary, so this cannot panic on well formed input. It is written as a slice rather than a checked get because a token whose bounds are not on a boundary is a bug in the tokenizer and silently returning nothing would hide it.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Token

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Token

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token