pub enum TokenKind {
Show 74 variants
Int(i64),
Float(f64),
Byte(u8),
Str(String),
StrStart(String),
StrMid(String),
StrEnd(String),
InterpStart,
InterpEnd,
Ident(String),
Fn,
Let,
Mut,
If,
Else,
While,
For,
In,
Return,
Break,
Continue,
Defer,
Match,
Struct,
Enum,
Interface,
Comptime,
Is,
Pure,
Io,
Alloc,
Panic,
Or,
SelfKw,
True,
False,
I64Ty,
F64Ty,
BoolTy,
StrTy,
ByteTy,
VoidTy,
Plus,
Minus,
Star,
Slash,
Percent,
EqEq,
BangEq,
Lt,
LtEq,
Gt,
GtEq,
AmpAmp,
PipePipe,
Bang,
Eq,
Dot,
Comma,
Colon,
Semi,
LParen,
RParen,
LBracket,
RBracket,
LBrace,
RBrace,
Arrow,
FatArrow,
PipeGt,
Question,
DotDot,
DotDotEq,
Eof,
}Expand description
what a token is. literals and identifiers carry their decoded value; every keyword, every operator, every piece of punctuation, and end of file are nullary variants.
PartialEq but not Eq, because TokenKind::Float holds an f64 and
f64 is not Eq. that is fine; nothing needs TokenKind: Eq.
Variants§
Int(i64)
an integer literal, already parsed (decimal, 0x, or 0b; underscores
stripped). stored as the non-negative magnitude; a leading - is a
separate token the parser folds in.
Float(f64)
a float literal, already parsed (underscores stripped, exponent applied).
Byte(u8)
a byte literal b'X', the single byte it denotes.
Str(String)
a string literal with no interpolation, escapes already decoded.
StrStart(String)
the decoded text before the first interpolation in a string with
interpolations (may be empty). followed by an TokenKind::InterpStart.
StrMid(String)
the decoded text between one interpolation’s } and the next one’s {
(may be empty). sits between an TokenKind::InterpEnd and an
TokenKind::InterpStart.
StrEnd(String)
the decoded text after the last interpolation in a string (may be empty).
follows an TokenKind::InterpEnd and closes the string.
InterpStart
opens an embedded interpolation expression. between this and the matching
TokenKind::InterpEnd the lexer emits ordinary tokens, so any
expression (including a nested string) works inside { ... }.
InterpEnd
closes the interpolation expression opened by TokenKind::InterpStart.
Ident(String)
an identifier: ASCII [A-Za-z_][A-Za-z0-9_]* that is not a keyword.
Fn
Let
Mut
If
Else
While
For
In
Return
Break
Continue
Defer
Match
Struct
Enum
Interface
Comptime
Is
Pure
Io
Alloc
Panic
Or
SelfKw
the self keyword. named SelfKw because Self and self are reserved
in Rust and cannot be used as identifiers here.
True
the boolean literal true. lexed as a keyword, not an identifier.
False
the boolean literal false. lexed as a keyword, not an identifier.
I64Ty
the primitive type name i64.
F64Ty
the primitive type name f64.
BoolTy
the primitive type name bool.
StrTy
the primitive type name str.
ByteTy
the primitive type name byte.
VoidTy
the primitive type name void.
Plus
+
Minus
-
Star
*
Slash
/
Percent
%
EqEq
==
BangEq
!=
Lt
<
LtEq
<=
Gt
>
GtEq
>=
AmpAmp
&&
PipePipe
||
Bang
!
Eq
=
Dot
.
Comma
,
Colon
:
Semi
;
LParen
(
RParen
)
LBracket
[
RBracket
]
LBrace
{
RBrace
}
Arrow
-> (return-type arrow)
FatArrow
=> (match-arm arrow)
PipeGt
|> (pipeline)
Question
? (postfix error propagation)
DotDot
.. (exclusive range)
DotDotEq
..= (inclusive range)
Eof
end of the token stream.