Struct Token

pub struct Token<'a, K: TokenKind> {
    pub kind: K,
    pub start: usize,
    pub end: usize,
    pub text: &'a str,
    pub skip: bool,
    pub next: Option<Box<Token<'a, K>>>,
    pub skipped: Vec<Token<'a, K>>,
    pub captures: Option<Vec<Option<(usize, usize)>>>,
}

Represents a detected token

Fields

kind: K

start: usize

end: usize

text: &'a str

skip: bool

next: Option<Box<Token<'a, K>>>

In some cases (viz. when an “unrecognized” token is returned), whatever token comes next is cached here.

skipped: Vec<Token<'a, K>>

The tokens skipped over to get here.

captures: Option<Vec<Option<(usize, usize)>>>

Implementations

impl<'a, K: TokenKind> Token<'a, K>

pub fn new(source: &'a str, kind: K, start: usize, end: usize) -> Token<'a, K>

pub fn from_text(kind: K, text: &'a str, start: usize) -> Token<'a, K>

pub fn eof(source: &'a str) -> Token<'a, K>

pub fn skip(self, skip: bool) -> Self

pub fn skipped(self, skipped: Vec<Token<'a, K>>) -> Self

pub fn next(self, next: Option<Box<Token<'a, K>>>) -> Self

pub fn captures(self, captures: Option<Vec<Option<(usize, usize)>>>) -> Self

pub fn get_capture(&self, idx: usize) -> Option<TokenCapture<'a>>

Trait Implementations

impl<'a, K: Debug + TokenKind> Debug for Token<'a, K>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a, K: PartialEq + TokenKind> PartialEq for Token<'a, K>

fn eq(&self, other: &Token<'a, K>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'a, K: TokenKind> StructuralPartialEq for Token<'a, K>