Enum TokenKind 

pub enum TokenKind {
    Number,
    Word,
    Quoted,
    Ip,
    Url,
    Email,
    Timestamp,
    Version,
    Uuid,
    Mac,
    HexColor,
    Cidr,
    Percent,
    ByteSize,
    Money,
    HashDigest,
    Hex,
    Duration,
    Path,
    Jwt,
    CreditCard,
    Base64,
    Geo,
    Phone,
    Quantity,
    Whitespace,
    Punct,
    Open(BracketKind),
    Close(BracketKind),
    Custom(u8),
    Other,
}

The typed class of a single token.

Each variant is the target of one surface atom. Promoting a span to one of these classes is the lexer’s job; once classed, a whole number or a whole quoted string is one atom, which is what lets a trex pattern stay on one line.

Variants

Number

Integer or floating-point literal. Surface atom: \N.

Word

Word or identifier run. Surface atom: \W.

Quoted

Quoted string, including its delimiters, with escapes resolved by the lexer. A string closes on its own line, or a backslash carries it past the newline; a quote its line does not close is punctuation. Surface atom: \Q.

Ip

IPv4 or IPv6 address. Surface atom: \I.

Url

URL. Surface atom: \U.

Email

Email address. Surface atom: \E.

Timestamp

Date or timestamp. Surface atom: \T.

Version

A semantic-version string, MAJOR.MINOR.PATCH with an optional -prerelease and/or +build suffix. Surface atom: \V.

Uuid

A UUID / GUID: 8-4-4-4-12 hex digits. Surface atom: \{uuid}.

Mac

A MAC / EUI-48 hardware address: six :- or --separated hex pairs. Surface atom: \A.

HexColor

A hex color literal, #rgb or #rrggbb. Surface atom: \H.

Cidr

A CIDR block: an IPv4 address with a /prefix. Surface atom: \C.

Percent

A percentage, N% or N.N%. Surface atom: \%.

ByteSize

A byte size with a unit, 10MB / 1.5GiB / 512KB. Surface atom: \Z.

Money

A money amount, $1,234.56 / $5. Surface atom: \$.

HashDigest

A hash digest: a run of exactly 32 / 40 / 64 hex characters with at least one hex letter (md5 / sha1 / sha256). Surface atom: \D.

Hex

A hex run: 32 or more hex characters with at least one hex letter, at a length no hash digest has (a key, a dump, a digest of another size). Surface atom: \{hex}.

Duration

A duration: one or more number + time-unit segments (1500ms, 2.5s, 3h20m). Surface atom: \R.

Path

A filesystem path (/usr/bin/x, ./rel, ../up, ~/home, C:\dir\file, \\server\share). Surface atom: \L.

Jwt

A JSON Web Token: three base64url segments separated by dots (header.payload.signature). Surface atom: \{jwt}.

CreditCard

A payment-card number, 13-19 digits, contiguous or grouped the way an issuer prints them (4-4-4-4, 4-6-5, 4-6-4 or 4-4-4-4-3, with one separator throughout, a space or a hyphen), that passes the Luhn check. Surface atom: \{creditcard}.

Base64

A base64 / base64url blob: a [A-Za-z0-9+/] run of length a multiple of four and at least 16, with charset diversity (so a plain word is not one) and optional = padding. Surface atom: \{base64}.

Geo

A geographic coordinate in decimal degrees, lat,long with both in range (lat -90..90, long -180..180), each carrying four fractional digits, and the pair on its own rather than inside a longer comma-separated run. Surface atom: \{geo}.

Phone

A telephone number: international, a + then an ITU-T E.164 country calling code and 7-15 digits in all; or North American written nationally, NPA-NXX-XXXX hyphenated with an optional 1- prefix. Surface atom: \{phone}.

Quantity

A physical quantity: a number, signed where nothing alphanumeric precedes the sign, then a unit symbol of crate::quantity’s table, attached or one space apart (5kg, -40°C, 3.2 GHz, 40 %). Surface atom: \{quantity}; \{qty} is the class of every kind a quantity predicate reads.

Whitespace

Run of insignificant whitespace. Surface atom: \S. Skipped between atoms in token-mode unless matched explicitly.

Punct

A single punctuation token. Surface atom: \P.

Open(BracketKind)

An opening bracket of the given kind. Its Token::mate points at the matching close.

Close(BracketKind)

A closing bracket of the given kind. Its Token::mate points back at the matching open.

Custom(u8)

A span matching a user-declared shape, identified by its id in the crate::custom::ShapeSet that lexed it. Surface atom: \{name}, the name that shape was declared under.

Other

A token the lexer did not assign a more specific class.

Implementations

impl TokenKind

pub fn is_insignificant(self) -> bool

Whether this kind is skipped between atoms in token-mode.

pub fn bracket_of_code(code: u32) -> Option<(bool, BracketKind)>

The bracket a kind code stands for and whether it opens, or None for a code that is no bracket.

Beside TokenKind::code so both directions of the encoding read from the one table, and derived from it rather than written out a second time. A reader holding codes and not kinds - the significant stream stores codes - needs this to match a close against the open it wants.

pub fn code(self) -> u32

A small, total, distinct integer code for this kind, used to carry the token-kind stream and the atom-kind table to a backend that cannot hold the Rust enum (the GPU kernel). The bracket kind is folded into the code so Open(Paren) and Open(Square) stay distinct.

pub fn from_code(code: u32) -> TokenKind

The kind a code names: the inverse of Self::code on every kind.

Panics

On a code no kind has, which only a stream written by something other than Self::code could carry.

impl TokenKind

One lexed token: a typed class plus the half-open byte span [start, end) it covers in the input.

The bracket-pairing result rides on the token: for an TokenKind::Open / TokenKind::Close token Token::mate is the index of the partner token in the stream, and None for every other kind.

A whole input’s tokens are one flat vector that every consumer streams, so the struct’s width is bandwidth. The mate is stored as the partner’s index plus one in a NonZeroU32, which puts the None in the zero niche and costs four bytes rather than the sixteen an Option<usize> takes.

pub fn name(self) -> &'static str

The kind’s name, as \{name} writes it in a pattern.

pub fn named(name: &str) -> Option<TokenKind>

The kind written under name, the inverse of Self::name for the kinds a name reaches.

A bracket is named by its side rather than its shape, since that is what Self::name reports; a declared kind has no name here, because the name belongs to the declaration rather than to the enum.

pub fn escape(self) -> Option<char>

The single letter that names the kind as \X in a pattern, for the kinds that have one.

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenKind

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenKind

impl Hash for TokenKind

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for TokenKind

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenKind