Enum Token 

pub enum Token<S> {
    Word(S),
    Num(S),
    Str(S, Vec<StrPart<S, Self>>),
    Op(S),
    Char(S),
    Block(S, Vec<Self>),
}

Token (tree) generic over string type S.

Variants

Word(S)

keywords such as def, but also identifiers such as map, $x, or @csv

Num(S)

number

Str(S, Vec<StrPart<S, Self>>)

(interpolated) string, surrounded by opening and closing ‘“’

Op(S)

binary operator, such as | or +=

Note that this includes - (negation) also when it is used as unary operator.

Char(S)

punctuation, such as . or ;

Block(S, Vec<Self>)

delimited tokens, e.g. (...) or [...]

Implementations

impl<'a> Token<&'a str>

pub fn opt_as_str(found: Option<&Self>, code: &'a str) -> &'a str

Return the string slice corresponding to an optional token.

If the token is not present, return an empty string slice starting at the end of code.

pub fn as_str(&self) -> &'a str

Return the string slice corresponding to the token.

pub fn span(&self, code: &str) -> Span

Return the span of a token that was lexed from some given input.

Trait Implementations

impl<S: Debug> Debug for Token<S>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.