Enum UniversalTokenRole 

pub enum UniversalTokenRole {
    Keyword,
    Name,
    Literal,
    Escape,
    Operator,
    Punctuation,
    Comment,
    Whitespace,
    Error,
    None,
    Eof,
}

Represents the general syntactic role of a token across diverse languages.

Universal Grammar

This mechanism is inspired by Noam Chomsky’s Universal Grammar theory. It posits that while the “Surface Structure” (specific token kinds) of languages may vary wildly, they share a common “Deep Structure” (syntactic roles).

In the Oak framework:

• Surface Structure: Refers to specific token kinds defined by a language (e.g., Rust’s PubKeyword).
• Deep Structure: Refers to the universal roles defined in this enum (e.g., UniversalTokenRole::Keyword).

By mapping to these roles, generic tools can identify names, literals, or operators across 100+ languages without needing to learn the specifics of each grammar.

Variants

Keyword

Language reserved words or built-in commands (e.g., ‘SELECT’, ‘let’, ‘MOV’).

Name

Identifiers, labels, keys, tags, or any name-like token.

Literal

Literal values like strings, numbers, booleans, or nulls.

Escape

An escape sequence or a special character representation within a literal.

Operator

Mathematical, logical, or structural operators (e.g., ‘+’, ‘=>’, ‘LIKE’).

Punctuation

Structural characters like brackets, commas, semicolons.

Comment

Developer annotations or documentation.

Whitespace

Formatting characters like spaces or tabs.

Error

Malformed or unrecognized content.

None

No specific role assigned.

Eof

End of stream marker.

Trait Implementations

impl Clone for UniversalTokenRole

fn clone(&self) -> UniversalTokenRole

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UniversalTokenRole

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for UniversalTokenRole

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Hash for UniversalTokenRole

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for UniversalTokenRole

fn eq(&self, other: &UniversalTokenRole) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for UniversalTokenRole

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl TokenRole for UniversalTokenRole

fn universal(&self) -> UniversalTokenRole

Maps this role to a universal, language-agnostic role.

fn name(&self) -> &str

Returns a specific name for this role, used for granular highlighting.

impl Copy for UniversalTokenRole

impl Eq for UniversalTokenRole

impl StructuralPartialEq for UniversalTokenRole