Enum TokenKind 

pub enum TokenKind {
    IsoStart,
    IsoEnd,
    Header,
    Data,
    EndSec,
    LParen,
    RParen,
    Comma,
    Semicolon,
    Equals,
    Asterisk,
    Dollar,
    Real(f64),
    Integer(i64),
    String(String),
    EntityRef(u64),
    Enum(String),
    Binary(String),
    Keyword(String),
    AnchorRef(String),
}

Part 21 token kinds.

Variants cover every lexical construct defined in ISO 10303-21.

Variants

IsoStart

IsoEnd

Header

Data

EndSec

LParen

RParen

Comma

Semicolon

Equals

Asterisk

Derived attribute marker.

Dollar

Unset / optional attribute marker.

Real(f64)

Real number — must contain a .; captures optional sign and exponent.

Integer(i64)

Integer — no decimal point; may carry a sign.

String(String)

String literal enclosed in single quotes. Outer quotes are stripped and the '' escape is decoded to a single '. Part 21 wide-char escapes (\X\HH, \X2\…\X0\, \X4\…\X0\) remain raw — decoding those is deferred to a later stage.

EntityRef(u64)

Entity reference (#N). The numeric identifier is stored as u64.

Enum(String)

Enumeration literal such as .T. or .MILLI.. The inner name is stored without the surrounding dots.

Binary(String)

Hex-encoded binary literal, stored without the surrounding quotes. The leading hex digit (0–3) encodes the number of unused bits.

Keyword(String)

Identifier / entity type name. Does not include -, so hyphenated section markers (handled above) never match this variant.

AnchorRef(String)

P21 edition 3 anchor reference (<name> or <url#name>). The whole bracketed form is captured as a single token; the parser currently discards these (ed.3 sections are skip-only — see crate::parser::ParseWarning::Ed3SectionDiscarded).

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'s> Logos<'s> for TokenKind

type Error = ()

Error type returned by the lexer. This can be set using #[logos(error = MyError)]. Defaults to () if not set.

type Extras = ()

Associated type Extras for the particular lexer. This can be set using #[logos(extras = MyExtras)] and accessed inside callbacks.

type Source = str

Source type this token can be lexed from. This will default to str, unless one of the defined patterns explicitly uses non-unicode byte values or byte slices, in which case that implementation will use [u8].

fn lex( lex: &mut Lexer<'s, Self>, ) -> Option<Result<Self, <Self as Logos<'s>>::Error>>

The heart of Logos. Called by the Lexer. The implementation for this function is generated by the logos-derive crate.

fn lexer(source: &'source Self::Source) -> Lexer<'source, Self>

where Self::Extras: Default,

Create a new instance of a Lexer that will produce tokens implementing this Logos.

fn lexer_with_extras( source: &'source Self::Source, extras: Self::Extras, ) -> Lexer<'source, Self>

Create a new instance of a Lexer with the provided Extras that will produce tokens implementing this Logos.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TokenKind