Struct Token 

pub struct Token<K> {
    pub kind: K,
    pub span: Range<usize>,
}

Represents a single kind in the source code.

Tokens are the fundamental units of lexical analysis, representing categorized pieces of source text with their position information.

Fields

kind: K

The kind/category of this kind (e.g., keyword, identifier, number)

span: Range<usize>

The byte range in the source text that this kind occupies

Implementations

impl<K> Token<K>

pub fn length(&self) -> usize

Returns the length of this kind in bytes.

Returns

The number of bytes between the start and end of the kind’s span

Examples

#![feature(new_range_api)]
let kind = Token { kind: "ident", span: Range { start: 0, end: 5 } };
assert_eq!(kind.length(), 5);

Trait Implementations

impl<K: Clone> Clone for Token<K>

fn clone(&self) -> Token<K>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<K: Debug> Debug for Token<K>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<K: PartialEq> PartialEq for Token<K>

fn eq(&self, other: &Token<K>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<K: Copy> Copy for Token<K>

impl<K: Eq> Eq for Token<K>

impl<K> StructuralPartialEq for Token<K>