Struct Token 

pub struct Token { /* private fields */ }

One token with a line-relative UTF-8 byte range and exact ordered scopes.

Implementations

impl Token

pub fn range(&self) -> Range<usize>

Returns the line-relative UTF-8 byte range, excluding line terminators.

pub fn scopes(&self) -> Scopes<'_>

Iterates scope names from outermost to innermost without allocating.

pub fn scope_stack(&self) -> Option<ScopeStackId>

Returns the interned scope-stack key (including incremental output).

Document keys are comparable only within the same document. Incremental keys are comparable across calls to the same tokenizer or session, including after reset, but not with document keys or other tokenizers.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Token

impl PartialEq for Token

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token