Enum TokenKind 

pub enum TokenKind {
    My,
    Our,
    Local,
    State,
    Sub,
    If,
    Elsif,
    Else,
    Unless,
    While,
    Until,
    For,
    Foreach,
    Return,
    Package,
    Use,
    No,
    Begin,
    End,
    Check,
    Init,
    Unitcheck,
    Eval,
    Do,
    Given,
    When,
    Default,
    Try,
    Catch,
    Finally,
    Continue,
    Next,
    Last,
    Redo,
    Goto,
    Class,
    Method,
    Field,
    Format,
    Undef,
    Defer,
    Assign,
    Plus,
    Minus,
    Star,
    Slash,
    Percent,
    Power,
    LeftShift,
    RightShift,
    BitwiseAnd,
    BitwiseOr,
    BitwiseXor,
    BitwiseNot,
    PlusAssign,
    MinusAssign,
    StarAssign,
    SlashAssign,
    PercentAssign,
    DotAssign,
    AndAssign,
    OrAssign,
    XorAssign,
    PowerAssign,
    LeftShiftAssign,
    RightShiftAssign,
    LogicalAndAssign,
    LogicalOrAssign,
    DefinedOrAssign,
    Equal,
    NotEqual,
    Match,
    NotMatch,
    SmartMatch,
    Less,
    Greater,
    LessEqual,
    GreaterEqual,
    Spaceship,
    StringCompare,
    And,
    Or,
    Not,
    DefinedOr,
    WordAnd,
    WordOr,
    WordNot,
    WordXor,
    Arrow,
    FatArrow,
    Dot,
    Range,
    Ellipsis,
    Increment,
    Decrement,
    DoubleColon,
    Question,
    Colon,
    Backslash,
    LeftParen,
    RightParen,
    LeftBrace,
    RightBrace,
    LeftBracket,
    RightBracket,
    Semicolon,
    Comma,
    Number,
    String,
    Regex,
    Substitution,
    Transliteration,
    QuoteSingle,
    QuoteDouble,
    QuoteWords,
    QuoteCommand,
    HeredocStart,
    HeredocBody,
    FormatBody,
    DataMarker,
    DataBody,
    VString,
    UnknownRest,
    HeredocDepthLimit,
    Identifier,
    ScalarSigil,
    ArraySigil,
    HashSigil,
    SubSigil,
    GlobSigil,
    Eof,
    Unknown,
}

Token classification for Perl parsing.

The set is intentionally simplified for fast parser matching while covering keywords, operators, delimiters, literals, identifiers, and special tokens.

Use TokenKind::display_name to get a human-readable string suitable for error messages shown to the user.

Categories

Group	Examples
Keywords	My, Sub, If, …
Operators	Plus, Arrow, And, …
Delimiters	LeftParen, LeftBrace, …
Literals	Number, String, Regex, …
Identifiers	Identifier, ScalarSigil, …
Special	Eof, Unknown

Variants

My

Lexical variable declaration: my $x

Our

Package variable declaration: our $x

Local

Dynamic scoping: local $x

State

Persistent variable: state $x

Sub

Subroutine declaration: sub foo

If

Conditional: if (cond)

Elsif

Else-if conditional: elsif (cond)

Else

Else branch: else { }

Unless

Negated conditional: unless (cond)

While

While loop: while (cond)

Until

Until loop: until (cond)

For

C-style for loop: for (init; cond; update)

Foreach

Iterator loop: foreach $x (@list)

Return

Return statement: return $value

Package

Package declaration: package Foo

Use

Module import: use Module

No

Disable pragma/module: no strict

Begin

Compile-time block: BEGIN { }

End

Exit-time block: END { }

Check

Check phase block: CHECK { }

Init

Init phase block: INIT { }

Unitcheck

Unit check block: UNITCHECK { }

Eval

Exception handling: eval { }

Do

Block execution: do { } or do "file"

Given

Switch expression: given ($x)

When

Case clause: when ($pattern)

Default

Default case: default { }

Try

Try block: try { }

Catch

Catch block: catch ($e) { }

Finally

Finally block: finally { }

Continue

Continue block: continue { }

Next

Loop control: next

Last

Loop control: last

Redo

Loop control: redo

Goto

Goto statement: goto LABEL, goto &sub, goto EXPR

Class

Class declaration (5.38+): class Foo

Method

Method declaration (5.38+): method foo

Field

Class field declaration (5.38+): field $name

Format

Format declaration: format STDOUT =

Undef

Undefined value: undef

Defer

Defer block: defer { ... } (Perl 5.36+ experimental, stable in 5.40)

Assign

Assignment: =

Plus

Addition: +

Minus

Subtraction: -

Star

Multiplication: *

Slash

Division: /

Percent

Modulo: %

Power

Exponentiation: **

LeftShift

Left bit shift: <<

RightShift

Right bit shift: >>

BitwiseAnd

Bitwise AND: &

BitwiseOr

Bitwise OR: |

BitwiseXor

Bitwise XOR: ^

BitwiseNot

Bitwise NOT: ~

PlusAssign

Add and assign: +=

MinusAssign

Subtract and assign: -=

StarAssign

Multiply and assign: *=

SlashAssign

Divide and assign: /=

PercentAssign

Modulo and assign: %=

DotAssign

Concatenate and assign: .=

AndAssign

Bitwise AND and assign: &=

OrAssign

Bitwise OR and assign: |=

XorAssign

Bitwise XOR and assign: ^=

PowerAssign

Power and assign: **=

LeftShiftAssign

Left shift and assign: <<=

RightShiftAssign

Right shift and assign: >>=

LogicalAndAssign

Logical AND and assign: &&=

LogicalOrAssign

Logical OR and assign: ||=

DefinedOrAssign

Defined-or and assign: //=

Equal

Numeric equality: ==

NotEqual

Numeric inequality: !=

Match

Pattern match binding: =~

NotMatch

Negated pattern match: !~

SmartMatch

Smart match: ~~

Less

Less than: <

Greater

Greater than: >

LessEqual

Less than or equal: <=

GreaterEqual

Greater than or equal: >=

Spaceship

Numeric comparison (spaceship): <=>

StringCompare

String comparison: cmp

And

Logical AND: &&

Or

Logical OR: ||

Not

Logical NOT: !

DefinedOr

Defined-or: //

WordAnd

Word AND operator: and

WordOr

Word OR operator: or

WordNot

Word NOT operator: not

WordXor

Word XOR operator: xor

Arrow

Method/dereference arrow: ->

FatArrow

Hash key separator: =>

Dot

String concatenation: .

Range

Range operator: ..

Ellipsis

Yada-yada (unimplemented): ...

Increment

Increment: ++

Decrement

Decrement: --

DoubleColon

Package separator: ::

Question

Ternary condition: ?

Colon

Ternary/label separator: :

Backslash

Reference operator: \

LeftParen

Left parenthesis: (

RightParen

Right parenthesis: )

LeftBrace

Left brace: {

RightBrace

Right brace: }

LeftBracket

Left bracket: [

RightBracket

Right bracket: ]

Semicolon

Statement terminator: ;

Comma

List separator: ,

Number

Numeric literal: 42, 3.14, 0xFF

String

String literal: "hello" or 'world'

Regex

Regular expression: /pattern/flags

Substitution

Substitution: s/pattern/replacement/flags

Transliteration

Transliteration: tr/abc/xyz/ or y///

QuoteSingle

Single-quoted string: q/text/

QuoteDouble

Double-quoted string: qq/text/

QuoteWords

Quote words: qw(list of words)

QuoteCommand

Backtick command: `cmd` or qx/cmd/

HeredocStart

Heredoc start marker: <<EOF

HeredocBody

Heredoc content body

FormatBody

Format specification body

DataMarker

Data section marker: __DATA__ or __END__

DataBody

Data section content

VString

Version string literal: v5.26.0, v5.10

UnknownRest

Unparsed remainder (budget exceeded)

HeredocDepthLimit

Heredoc depth limit exceeded (special error token)

Identifier

Bareword identifier or function name

ScalarSigil

Scalar sigil: $

ArraySigil

Array sigil: @

HashSigil

Hash sigil: %

SubSigil

Subroutine sigil: &

GlobSigil

Glob/typeglob sigil: *

Eof

End of file/input

Unknown

Unknown/unrecognized token

Implementations

impl TokenKind

pub const fn all() -> &'static [TokenKind]

Return every TokenKind variant in stable declaration order.

pub const fn metadata_count() -> usize

Number of token kinds expected to have metadata coverage.

pub fn metadata(self) -> TokenKindMetadata

Return compact metadata for this token kind.

pub const fn category(self) -> TokenCategory

Return the high-level category for this token kind.

pub const fn is_keyword(self) -> bool

Returns true if this token kind is a keyword.

pub const fn is_operator(self) -> bool

Returns true if this token kind is an operator.

pub const fn is_literal(self) -> bool

Returns true if this token kind is a literal.

pub const fn is_delimiter(self) -> bool

Returns true if this token kind is a delimiter.

pub const fn is_identifier(self) -> bool

Returns true if this token kind is an identifier or sigil.

pub const fn is_special(self) -> bool

Returns true if this token kind is a special sentinel/recovery token.

pub fn is_assignment_operator(self) -> bool

Return whether this token is an assignment operator.

pub fn is_comparison_operator(self) -> bool

Return whether this token is a comparison operator.

pub fn is_logical_operator(self) -> bool

Return whether this token is a logical operator.

pub fn is_word_operator(self) -> bool

Return whether this token is a word-form operator token.

pub fn is_low_precedence_word_operator(self) -> bool

Return whether this token is a low-precedence word operator.

pub fn is_open_delimiter(self) -> bool

Return whether this token is an opening paired delimiter.

pub fn is_close_delimiter(self) -> bool

Return whether this token is a closing paired delimiter.

pub fn matching_delimiter(self) -> Option<Self>

Return the matching paired delimiter for this token, if any.

pub fn is_quote_like(self) -> bool

Return whether this token is quote-like syntax.

pub fn is_recovery_boundary(self) -> bool

Return whether this token is a hard recovery boundary.

pub fn from_keyword(spelling: &str) -> Option<TokenKind>

Map a canonical keyword spelling to its TokenKind.

This mapping is case-sensitive and only recognizes canonical Perl spellings used by the lexer/parser pipeline.

pub fn from_operator(spelling: &str) -> Option<TokenKind>

Map a canonical operator spelling to its TokenKind.

This mapping is case-sensitive.

pub fn from_delimiter(spelling: &str) -> Option<TokenKind>

Map a delimiter spelling to its TokenKind.

pub fn from_sigil(spelling: &str) -> Option<TokenKind>

Map a sigil spelling to its TokenKind.

pub fn display_name(self) -> &'static str

Return a user-friendly display name for this token kind.

These names appear in parser error messages shown in the editor. They use the actual Perl syntax (e.g. } instead of RightBrace) so users can immediately understand what the parser expected.

Examples

use perl_token::TokenKind;

assert_eq!(TokenKind::Semicolon.display_name(), "';'");
assert_eq!(TokenKind::Sub.display_name(), "'sub'");
assert_eq!(TokenKind::Number.display_name(), "number");

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralPartialEq for TokenKind