Enum TokenType 

pub enum TokenType {
    AsciiWord = 1,
    Word = 2,
    NumWord = 3,
    Email = 4,
    Url = 5,
    Host = 6,
    SFloat = 7,
    Version = 8,
    HwordNumPart = 9,
    HwordPart = 10,
    HwordAsciiPart = 11,
    Blank = 12,
    Tag = 13,
    Protocol = 14,
    NumHword = 15,
    AsciiHword = 16,
    Hword = 17,
    UrlPath = 18,
    File = 19,
    Float = 20,
    Int = 21,
    Uint = 22,
    Entity = 23,
}

Tokenise on Unicode word boundaries — anything that is not an alphanumeric scalar value (or _) splits the token. Lowercases each emitted token. v7.39 (round 651) — PG’s token types, as ts_token_type('default') publishes them. Only the ones SPG’s parser actually produces are here; the numbering is PG’s so pg_ts_config_map.maptokentype and ts_debug.alias agree with it.

The four PG does NOT map to any dictionary — blank(12), tag(13), protocol(14), entity(23) — are recognised precisely so they can be DROPPED. That is the difference between indexing <b>x</b> as x, which PG does, and as b, x, b, which SPG did.

Variants

AsciiWord = 1

Word = 2

NumWord = 3

Email = 4

Url = 5

Host = 6

SFloat = 7

Version = 8

HwordNumPart = 9

HwordPart = 10

HwordAsciiPart = 11

Blank = 12

Tag = 13

Protocol = 14

NumHword = 15

AsciiHword = 16

Hword = 17

UrlPath = 18

File = 19

Float = 20

Int = 21

Uint = 22

Entity = 23

Implementations

impl TokenType

pub const fn alias(self) -> &'static str

PG’s alias column.

pub const fn description(self) -> &'static str

PG’s description column, verbatim.

pub const fn dictionary(self, english: bool) -> Option<TsDict>

Which dictionary a configuration sends this token to, or None when the configuration maps it to nothing and the token produces no lexeme at all. Read off PG18’s pg_ts_config_map: the same nineteen types are mapped by both simple and english, and the four that are not are blank, tag, protocol and entity.

Trait Implementations

impl Clone for TokenType

fn clone(&self) -> TokenType

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenType

impl Debug for TokenType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenType

impl PartialEq for TokenType

fn eq(&self, other: &TokenType) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenType