Enum UniversalTokenRole 

pub enum UniversalTokenRole {
    Keyword,
    Name,
    Literal,
    Escape,
    Operator,
    Punctuation,
    Comment,
    Whitespace,
    Error,
    None,
    Eof,
}

Represents the general syntactic role of a token across diverse languages.

By mapping to these roles, generic tools can identify names, literals, or operators across 100+ languages without needing to learn the specifics of each grammar.

Variants

Keyword

Language reserved words or built-in commands (e.g., ‘SELECT’, ‘let’, ‘MOV’).

Name

Identifiers, labels, keys, tags, or any name-like token.

Literal

Literal values like strings, numbers, booleans, or nulls.

Escape

An escape sequence or a special character representation within a literal.

Operator

Mathematical, logical, or structural operators (e.g., ‘+’, ‘=>’, ‘LIKE’).

Punctuation

Structural characters like brackets, commas, semicolons.

Comment

Developer annotations or documentation.

Whitespace

Formatting characters like spaces or tabs.

Error

Malformed or unrecognized content.

None

No specific role assigned.

Eof

End of stream marker.

Trait Implementations

impl Clone for UniversalTokenRole

fn clone(&self) -> UniversalTokenRole

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for UniversalTokenRole

impl Debug for UniversalTokenRole

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for UniversalTokenRole

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for UniversalTokenRole

impl Hash for UniversalTokenRole

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for UniversalTokenRole

fn eq(&self, other: &UniversalTokenRole) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for UniversalTokenRole

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for UniversalTokenRole

impl TokenRole for UniversalTokenRole

fn universal(&self) -> UniversalTokenRole

Maps this role to a universal, language-agnostic role.

fn name(&self) -> &str

Returns a specific name for this role, used for granular highlighting.