Enum Token 

#[non_exhaustive]
pub enum Token {
    Set,
    Local,
    If,
    Then,
    Else,
    Elif,
    Fi,
    For,
    While,
    In,
    Do,
    Done,
    Case,
    Esac,
    Function,
    Break,
    Continue,
    Return,
    Exit,
    True,
    False,
    TypeString,
    TypeInt,
    TypeFloat,
    TypeBool,
    And,
    Or,
    EqEq,
    NotEq,
    Match,
    NotMatch,
    GtEq,
    LtEq,
    GtGt,
    StderrToStdout,
    StdoutToStderr,
    StdoutToStderr2,
    Stderr,
    Both,
    HereString,
    HereDocStart,
    DoubleSemi,
    Eq,
    Pipe,
    Amp,
    Gt,
    Lt,
    Semi,
    Colon,
    Comma,
    DotDotDot,
    DotDot,
    Dot,
    TildePath(String),
    Tilde,
    RelativePath(String),
    DotSlashPath(String),
    DottedIdent(String),
    LBrace,
    RBrace,
    LBracket,
    RBracket,
    LParen,
    RParen,
    Star,
    Bang,
    Question,
    GlobWord(String),
    Arithmetic(String),
    CmdSubstStart,
    LongFlag(String),
    ShortFlag(String),
    PlusFlag(String),
    DoubleDash,
    DoubleDashBare(String),
    PlusBare(String),
    MinusBare(String),
    JobSpec(String),
    MinusAlone,
    String(String),
    SingleString(String),
    VarRef(String),
    SimpleVarRef(String),
    Positional(usize),
    AllArgs,
    ArgCount,
    LastExitCode,
    CurrentPid,
    VarLength(String),
    HereDoc(HereDocData),
    Int(i64),
    Float(f64),
    NumberIdent(String),
    DashNumWord(String),
    AtWord(String),
    InvalidFloatNoLeading,
    InvalidFloatNoTrailing,
    Path(String),
    Ident(String),
    Comment,
    Newline,
    LineContinuation,
    BacktickRejected,
}

A word is anything that is not whitespace and not an operator, so the bareword and path rules below admit \u{80}-\u{10FFFF} — this file’s spelling of “any non-ASCII scalar value” — alongside their ASCII classes. bash never inspects a word’s bytes for alphabetic-ness, and café, 日本語, and ~/文書 lex the same shape as their ASCII equivalents.

Variable names accept the same characters, and are NFC-normalized where the reference is built (VarPath::simple), so a name spelled with a combining mark and one spelled precomposed reach the same variable.

Flag names (LongFlag, ShortFlag, PlusFlag) are the exception and stay ASCII. --café is ambiguous — a flag no tool defines, or a word the caller meant literally — so kaish refuses rather than guessing, and the error says to quote it. Those rules still match a non-ASCII tail and reject it in their callback with LexerError::NonAsciiName; declining to match would split the word into a flag plus a stray bareword argument instead.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Set

Local

If

Then

Else

Elif

Fi

For

While

In

Do

Done

Case

Esac

Function

Break

Continue

Return

Exit

True

False

TypeString

TypeInt

TypeFloat

TypeBool

And

Or

EqEq

NotEq

Match

NotMatch

GtEq

LtEq

GtGt

StderrToStdout

StdoutToStderr

StdoutToStderr2

Stderr

Both

HereString

HereDocStart

DoubleSemi

Eq

Pipe

Amp

Gt

Lt

Semi

Colon

Comma

DotDotDot

Spread operator: [...$xs date]. Only meaningful inside a list literal (value context); inert everywhere else. logos resolves the "..." vs ".." (DotDot) ambiguity by longest match, so no explicit priority is needed here.

DotDot

Dot

TildePath(String)

Tilde path: ~/foo, ~user/bar - value includes the full string.

Tilde

Bare tilde: ~ alone (expands to $HOME)

RelativePath(String)

Relative path: ../foo/bar, bare src/kaish (ident containing /), or a directory reference with a trailing slash like dest/. The trailing-slash form uses * (not +) after the slash so dest/ lexes as one token instead of Ident("dest") + Path("/") — the latter split silently turned cp a b dest/ into a 4-operand command.

DotSlashPath(String)

Dot-slash path: ./foo, ./script.sh.

DottedIdent(String)

Dot-prefixed bareword: .parent, .gitignore, .foo.bar. Treated as an opaque string in argv position. Distinct from Token::Dot (the POSIX . source alias) which only matches a bare . — the source alias requires whitespace before its file argument (. script), so .parent (no space) is unambiguously a single bareword.

LBrace

RBrace

LBracket

RBracket

LParen

RParen

Star

Bang

Question

GlobWord(String)

Merged glob word: span-adjacent tokens containing *, ?, or [...]. Synthesized by merge_glob_adjacent(), never produced by logos directly.

Arithmetic(String)

Arithmetic expression content: synthesized by preprocessing. Contains the expression string between $(( and )).

CmdSubstStart

Command substitution start: $( - begins a command substitution

LongFlag(String)

Long flag: --name or --foo-bar. Flag names are ASCII-only; the match region still admits non-ASCII in the tail so the regex claims the WHOLE word instead of stopping at the ASCII prefix; without that, --café would lex as LongFlag(caf) plus a silently separate Ident(é) argument rather than one loud error. lex_long_flag rejects the match if it isn’t pure ASCII.

ShortFlag(String)

Short flag: -l, -la (combined short flags), or a dash-word with internal hyphens like -not-a-flag. Internal hyphens are part of the single shell word — without them the word fragments into separate flag tokens, which breaks echo -- -not-a-flag and the like. A leading -- is still DoubleDash (the second char must be a letter here) unless the third char isn’t a letter either, in which case it’s DoubleDashBare — see below — and whether the word is a flag or a literal is the binding layer’s call.

PlusFlag(String)

Plus flag: +e or +x (for set +e to disable options).

DoubleDash

Double dash: -- alone marks end of flags. Only matches when nothing else follows (a longer match always wins) — a ---prefixed word with more characters after it either lexes as LongFlag (3rd char is a letter) or DoubleDashBare (3rd char is anything else).

DoubleDashBare(String)

Bare word starting with -- whose continuation isn’t a valid long-flag name: ---, ----, --=x, --1, etc. Without this, the plain -- literal above always won the length tie against a lone --, silently truncating a dash-only operand to its trailing remainder (echo --- printed - instead of --- — GH #137). Mirrors MinusBare/PlusBare (bare-word fallback for an unrecognized flag-shaped prefix), just generalized to the -- prefix. A standalone -- (followed by whitespace/EOF) still lexes as DoubleDash — this regex requires at least one more non-whitespace character, so the two never tie in match length and no priority tiebreak is load-bearing; priority = 2 is set for consistency with PlusBare’s tier.

Both character classes exclude the unquoted shell operator characters ()|&;<> in addition to whitespace (GH #144): without that exclusion a case pattern like ---) swallowed the closing paren into the token text (DoubleDashBare("---)"), leaving no RParen for the branch parser to find — the same silent-truncation failure mode as #137, just on the other side of the word.

PlusBare(String)

Bare word starting with + followed by non-letter: +%s, +%Y-%m-%d For date format strings and similar. Lower priority than PlusFlag. See DoubleDashBare above for why ()|&;<> are excluded (GH #144).

MinusBare(String)

Bare word starting with - followed by non-letter/digit/dash: -%, etc. For rare cases. Lower priority than ShortFlag, Int, and DoubleDash. Excludes - after first - to avoid matching –name patterns. See DoubleDashBare above for why ()|&;<> are excluded (GH #144).

JobSpec(String)

Job specifier: %1, %2 — the bash idiom for wait/kill targets. Keeps the leading % (kill uses it to distinguish a job from a PID; wait strips it). Without this token a bare %1 is a lexer error.

MinusAlone

Standalone - (stdin indicator for cat -, diff - -, etc.) Only matches when followed by whitespace or end. This is handled specially in the parser as a positional arg.

String(String)

Double-quoted string: "..." — value is the parsed content (quotes removed, escapes processed). The regex matches only the opening quote; the callback extends the token to the quote that actually closes it, tracking $( depth so a quoted word INSIDE a substitution belongs to the substitution: "$(basename "$p")" is one string, not two. Same technique as VarRef below, for the same reason (GH #173).

SingleString(String)

Single-quoted string: '...' - literal content, no escape processing

VarRef(String)

Braced variable reference: ${VAR}, ${VAR.field}, or a default form with a NESTED reference like ${X:-${Y}} — value is the raw ${...} text. The regex matches only the ${ opener; the callback extends the token to the BALANCED closing brace (GH #173 — a plain [^}]+ regex stopped at the first } and split nested references). ${#VAR} still lexes as VarLength: its full regex out-matches this two-character opener, so logos selects it first.

SimpleVarRef(String)

Simple variable reference: $NAME - just the identifier. A name is ASCII alphanumerics, _, or any non-ASCII scalar value, so $café, $名前, and $😁 name variables the same way $NAME does. The name is NFC-normalized when the reference is built (VarPath::simple).

Positional(usize)

Positional parameter: $0 through $9

AllArgs

All positional parameters: $@

ArgCount

Number of positional parameters: $#

LastExitCode

Last exit code: $?

CurrentPid

Current shell PID: $$

VarLength(String)

Variable string length: ${#VAR} or a subscripted path ${#u[tags]}. The trailing (\[[^\]]*\])* admits chained bracket subscripts so a length-of-path lexes in expression position, not just inside strings; the parser turns the captured inner into a VarPath.

HereDoc(HereDocData)

Here-doc content: synthesized by preprocessing, not directly lexed. Contains the full content of the here-doc (without the delimiter lines).

Int(i64)

Integer literal - value is the parsed i64

Float(f64)

Float literal - value is the parsed f64

NumberIdent(String)

Digit-leading bareword: 019dda1c (SHA prefix), UUIDs, version-ish strings. Distinguished from Int because at least one alpha character follows the leading digits — the lexer commits to “this is a string, not a number.” Treated as a bareword string in expression position.

DashNumWord(String)

Numeric word containing an embedded hyphen run, or a minus-led numeric word with a non-numeric suffix. These are single contiguous shell words the user typed — ISO dates (2024-01-02), N-M ranges (10-20, cut -f 1-3, tr -d 0-9), float-dash forms (1.5-2), and find predicate values like -1k (smaller than 1k). Without this token they fragment into adjacent Int/Float/flag tokens and trip the no-token-pasting guard. The raw slice is preserved verbatim (so leading zeros survive). A plain 2024/1.5/-1 stays Int/Float — the digit-hyphen form requires a -segment, and the minus-led form requires an alpha after the digits.

AtWord(String)

Leading-@ bareword: @scope/pkg (scoped package), @0 (epoch in date -d @0), or bare @. Mid-word @ (user@host) is handled by Ident; this covers the leading-@ cases that would otherwise be an “unexpected character” lexer error.

InvalidFloatNoLeading

Invalid: float without leading digit (like .5)

InvalidFloatNoTrailing

Invalid: float without trailing digit (like 5.) Logos uses longest-match, so valid floats like 5.5 will match Float pattern instead

Path(String)

Absolute path: /tmp/out, /etc/hosts, /tmp/日本語, etc.

Ident(String)

Identifier - value is the identifier string Allows dots for filenames like script.kai and @ for user@host, a@b.com (bare @ is an ordinary word character, as in bash). The leading class excludes digits — NumberIdent/Int own digit-leading words — and the ASCII operator/whitespace set.

Comment

Comment: # ... to end of line, and only where a word can start.

# is an ordinary character inside a word — echo abc#3 prints abc#3, as it does in bash and sh — so the word classes above carry # and a mid-word # never reaches this rule. What does reach it after a non-word character is a loud error, not a comment: see lex_comment.

Newline

Newline (significant in kaish - ends statements)

LineContinuation

Line continuation: backslash at end of line

BacktickRejected

Backtick command substitution — explicitly rejected. Kaish drops backticks; the callback always errors so users get a dedicated BackticksNotSupported message instead of the generic UnexpectedCharacter they would have hit before. Backticks inside single/double-quoted strings, heredoc bodies, and comments don’t reach this match — those tokens are matched as a single unit (strings) or extracted before logos runs (heredocs) or skipped to EOL (comments).

Implementations

impl Token

pub fn category(&self) -> TokenCategory

Returns the semantic category for syntax highlighting.

impl Token

pub fn is_keyword(&self) -> bool

Returns true if this token is a keyword.

pub fn is_type(&self) -> bool

Returns true if this token is a type keyword.

pub fn starts_statement(&self) -> bool

Returns true if this token starts a statement.

pub fn is_value(&self) -> bool

Returns true if this token can appear in an expression.

Trait Implementations

impl Clone for Token

fn clone(&self) -> Token

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Token

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'s> Logos<'s> for Token

type Error = LexerError

Error type returned by the lexer. This can be set using #[logos(error = MyError)]. Defaults to () if not set.

type Extras = ()

Associated type Extras for the particular lexer. This can be set using #[logos(extras = MyExtras)] and accessed inside callbacks.

type Source = str

Source type this token can be lexed from. This will default to str, unless one of the defined patterns explicitly uses non-unicode byte values or byte slices, in which case that implementation will use [u8].

fn lex( lex: &mut Lexer<'s, Self>, ) -> Option<Result<Self, <Self as Logos<'s>>::Error>>

The heart of Logos. Called by the Lexer. The implementation for this function is generated by the logos-derive crate.

fn lexer(source: &'source Self::Source) -> Lexer<'source, Self>

where Self::Extras: Default,

Create a new instance of a Lexer that will produce tokens implementing this Logos.

fn lexer_with_extras( source: &'source Self::Source, extras: Self::Extras, ) -> Lexer<'source, Self>

Create a new instance of a Lexer with the provided Extras that will produce tokens implementing this Logos.

impl PartialEq for Token

fn eq(&self, other: &Token) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Token