Enum TokenKind 

pub enum TokenKind {
    My,
    Our,
    Local,
    State,
    Sub,
    If,
    Elsif,
    Else,
    Unless,
    While,
    Until,
    For,
    Foreach,
    Return,
    Package,
    Use,
    No,
    Begin,
    End,
    Check,
    Init,
    Unitcheck,
    Eval,
    Do,
    Given,
    When,
    Default,
    Try,
    Catch,
    Finally,
    Continue,
    Next,
    Last,
    Redo,
    Goto,
    Class,
    Method,
    Field,
    Format,
    Undef,
    Assign,
    Plus,
    Minus,
    Star,
    Slash,
    Percent,
    Power,
    LeftShift,
    RightShift,
    BitwiseAnd,
    BitwiseOr,
    BitwiseXor,
    BitwiseNot,
    PlusAssign,
    MinusAssign,
    StarAssign,
    SlashAssign,
    PercentAssign,
    DotAssign,
    AndAssign,
    OrAssign,
    XorAssign,
    PowerAssign,
    LeftShiftAssign,
    RightShiftAssign,
    LogicalAndAssign,
    LogicalOrAssign,
    DefinedOrAssign,
    Equal,
    NotEqual,
    Match,
    NotMatch,
    SmartMatch,
    Less,
    Greater,
    LessEqual,
    GreaterEqual,
    Spaceship,
    StringCompare,
    And,
    Or,
    Not,
    DefinedOr,
    WordAnd,
    WordOr,
    WordNot,
    WordXor,
    Arrow,
    FatArrow,
    Dot,
    Range,
    Ellipsis,
    Increment,
    Decrement,
    DoubleColon,
    Question,
    Colon,
    Backslash,
    LeftParen,
    RightParen,
    LeftBrace,
    RightBrace,
    LeftBracket,
    RightBracket,
    Semicolon,
    Comma,
    Number,
    String,
    Regex,
    Substitution,
    Transliteration,
    QuoteSingle,
    QuoteDouble,
    QuoteWords,
    QuoteCommand,
    HeredocStart,
    HeredocBody,
    FormatBody,
    DataMarker,
    DataBody,
    VString,
    UnknownRest,
    HeredocDepthLimit,
    Identifier,
    ScalarSigil,
    ArraySigil,
    HashSigil,
    SubSigil,
    GlobSigil,
    Eof,
    Unknown,
}

Token types and token stream for lexer output. Token classification for Perl parsing.

The set is intentionally simplified for fast parser matching while covering keywords, operators, delimiters, literals, identifiers, and special tokens.

Use TokenKind::display_name to get a human-readable string suitable for error messages shown to the user.

Categories

Group	Examples
Keywords	My, Sub, If, …
Operators	Plus, Arrow, And, …
Delimiters	LeftParen, LeftBrace, …
Literals	Number, String, Regex, …
Identifiers	Identifier, ScalarSigil, …
Special	Eof, Unknown

Variants

My

Lexical variable declaration: my $x

Our

Package variable declaration: our $x

Local

Dynamic scoping: local $x

State

Persistent variable: state $x

Sub

Subroutine declaration: sub foo

If

Conditional: if (cond)

Elsif

Else-if conditional: elsif (cond)

Else

Else branch: else { }

Unless

Negated conditional: unless (cond)

While

While loop: while (cond)

Until

Until loop: until (cond)

For

C-style for loop: for (init; cond; update)

Foreach

Iterator loop: foreach $x (@list)

Return

Return statement: return $value

Package

Package declaration: package Foo

Use

Module import: use Module

No

Disable pragma/module: no strict

Begin

Compile-time block: BEGIN { }

End

Exit-time block: END { }

Check

Check phase block: CHECK { }

Init

Init phase block: INIT { }

Unitcheck

Unit check block: UNITCHECK { }

Eval

Exception handling: eval { }

Do

Block execution: do { } or do "file"

Given

Switch expression: given ($x)

When

Case clause: when ($pattern)

Default

Default case: default { }

Try

Try block: try { }

Catch

Catch block: catch ($e) { }

Finally

Finally block: finally { }

Continue

Continue block: continue { }

Next

Loop control: next

Last

Loop control: last

Redo

Loop control: redo

Goto

Goto statement: goto LABEL, goto &sub, goto EXPR

Class

Class declaration (5.38+): class Foo

Method

Method declaration (5.38+): method foo

Field

Class field declaration (5.38+): field $name

Format

Format declaration: format STDOUT =

Undef

Undefined value: undef

Assign

Assignment: =

Plus

Addition: +

Minus

Subtraction: -

Star

Multiplication: *

Slash

Division: /

Percent

Modulo: %

Power

Exponentiation: **

LeftShift

Left bit shift: <<

RightShift

Right bit shift: >>

BitwiseAnd

Bitwise AND: &

BitwiseOr

Bitwise OR: |

BitwiseXor

Bitwise XOR: ^

BitwiseNot

Bitwise NOT: ~

PlusAssign

Add and assign: +=

MinusAssign

Subtract and assign: -=

StarAssign

Multiply and assign: *=

SlashAssign

Divide and assign: /=

PercentAssign

Modulo and assign: %=

DotAssign

Concatenate and assign: .=

AndAssign

Bitwise AND and assign: &=

OrAssign

Bitwise OR and assign: |=

XorAssign

Bitwise XOR and assign: ^=

PowerAssign

Power and assign: **=

LeftShiftAssign

Left shift and assign: <<=

RightShiftAssign

Right shift and assign: >>=

LogicalAndAssign

Logical AND and assign: &&=

LogicalOrAssign

Logical OR and assign: ||=

DefinedOrAssign

Defined-or and assign: //=

Equal

Numeric equality: ==

NotEqual

Numeric inequality: !=

Match

Pattern match binding: =~

NotMatch

Negated pattern match: !~

SmartMatch

Smart match: ~~

Less

Less than: <

Greater

Greater than: >

LessEqual

Less than or equal: <=

GreaterEqual

Greater than or equal: >=

Spaceship

Numeric comparison (spaceship): <=>

StringCompare

String comparison: cmp

And

Logical AND: &&

Or

Logical OR: ||

Not

Logical NOT: !

DefinedOr

Defined-or: //

WordAnd

Word AND operator: and

WordOr

Word OR operator: or

WordNot

Word NOT operator: not

WordXor

Word XOR operator: xor

Arrow

Method/dereference arrow: ->

FatArrow

Hash key separator: =>

Dot

String concatenation: .

Range

Range operator: ..

Ellipsis

Yada-yada (unimplemented): ...

Increment

Increment: ++

Decrement

Decrement: --

DoubleColon

Package separator: ::

Question

Ternary condition: ?

Colon

Ternary/label separator: :

Backslash

Reference operator: \

LeftParen

Left parenthesis: (

RightParen

Right parenthesis: )

LeftBrace

Left brace: {

RightBrace

Right brace: }

LeftBracket

Left bracket: [

RightBracket

Right bracket: ]

Semicolon

Statement terminator: ;

Comma

List separator: ,

Number

Numeric literal: 42, 3.14, 0xFF

String

String literal: "hello" or 'world'

Regex

Regular expression: /pattern/flags

Substitution

Substitution: s/pattern/replacement/flags

Transliteration

Transliteration: tr/abc/xyz/ or y///

QuoteSingle

Single-quoted string: q/text/

QuoteDouble

Double-quoted string: qq/text/

QuoteWords

Quote words: qw(list of words)

QuoteCommand

Backtick command: `cmd` or qx/cmd/

HeredocStart

Heredoc start marker: <<EOF

HeredocBody

Heredoc content body

FormatBody

Format specification body

DataMarker

Data section marker: __DATA__ or __END__

DataBody

Data section content

VString

Version string literal: v5.26.0, v5.10

UnknownRest

Unparsed remainder (budget exceeded)

HeredocDepthLimit

Heredoc depth limit exceeded (special error token)

Identifier

Bareword identifier or function name

ScalarSigil

Scalar sigil: $

ArraySigil

Array sigil: @

HashSigil

Hash sigil: %

SubSigil

Subroutine sigil: &

GlobSigil

Glob/typeglob sigil: *

Eof

End of file/input

Unknown

Unknown/unrecognized token

Implementations

impl TokenKind

pub fn display_name(self) -> &'static str

Return a user-friendly display name for this token kind.

These names appear in parser error messages shown in the editor. They use the actual Perl syntax (e.g. } instead of RightBrace) so users can immediately understand what the parser expected.

Examples

use perl_token::TokenKind;

assert_eq!(TokenKind::Semicolon.display_name(), "';'");
assert_eq!(TokenKind::Sub.display_name(), "'sub'");
assert_eq!(TokenKind::Number.display_name(), "number");

Trait Implementations

impl Clone for TokenKind

fn clone(&self) -> TokenKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl PartialEq for TokenKind

fn eq(&self, other: &TokenKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenKind

impl Eq for TokenKind

impl StructuralPartialEq for TokenKind