Enum Token 

pub enum Token {
    Punctuation,
    Keyword,
    Entity,
    Support,
    Constant,
    String,
    Comment,
    Invalid,
}

What a glyph in a fenced code block is to the language it is written in — the syntax-highlighting vocabulary, one level down from Role.

Eight classes rather than the hundreds of scopes a Sublime grammar names, and the same eight plates colours its published pages with: each is the first atom of a TextMate scope — keyword.control.rust is a keyword, string.quoted.double a string — and a palette that tells these eight apart is a palette that reads. Finer than this is a theme’s business, and a theme is exactly what core does not carry: like Role, a token says what a glyph is and leaves what colour to paint it to the frontend, so the same document highlights in ANSI colours on a terminal and in an Hsla in a GUI.

Only a glyph with Role::Code carries one, and only in a fenced block whose info string names a language the bundled grammars know (see crate::syntax). Inline `code`, an indented block, a bare fence, and a language no grammar covers all carry None, and draw in the code colour exactly as they did before this existed.

Variants

Punctuation

Brackets, operators, separators — punctuation.*. The quietest class: a frontend typically draws it in the comment colour, so the " opening a string reads as the string’s and not as its own thing.

Keyword

A reserved word or a storage modifier — keyword.* and storage.* (fn, let, pub, const, if).

Entity

A name the author defined or named — entity.* (a function or type at its definition) and variable.* (a parameter, a field, self).

Support

A name the language or its library provides — support.* (a builtin function, a standard type).

Constant

A literal that is not a string — constant.* (a number, true, an escape sequence, a character literal).

String

A string literal, delimiters included — string.*.

Comment

A comment — comment.*. A frontend typically italicises it as well.

Invalid

What the grammar could not parse — invalid.*.

Implementations

impl Token

pub const ALL: [Self; 8]

Every token, in precedence order: a scope whose atoms name two of these (punctuation.definition.string.begin is both punctuation and string) is the later one, so a string’s quotes read as string and a comment’s // as comment. The frontends iterate this to build their palettes, so a token added here is one a palette test immediately demands an entry for.

pub const fn name(self) -> &'static str

The class id a frontend keys a stylesheet or a palette on — the first atom of the scope it stands for, so a rule written for plates output (plates-keyword) and one written for leaf (leaf-t-keyword) name the same thing.

pub fn from_name(name: &str) -> Option<Self>

The token a class id names, or None for a name outside the vocabulary — the inverse of name.

pub const fn index(self) -> usize

This token’s position in ALL — the index a frontend’s own palette array is keyed by, the way MarkColor::index keys the highlighter washes.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Token

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Token

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token