pub enum Token {
Show 133 variants
HeaderRequire(String),
HeaderImport(String),
HeaderStage0,
HeaderStage1,
HeaderPersistent0,
Var(String),
VarWithMod(Vec<String>, String),
Constructor(String),
TypeVar(String),
OpenModule(String),
LongUpper(Vec<String>, String),
IntConst(i64),
FloatConst(f64),
LengthConst(f64, String),
Literal {
body: String,
omit_pre: bool,
omit_post: bool,
},
Mod,
If,
Then,
Else,
Let,
LetRec,
LetAnd,
In,
Fun,
True,
False,
Before,
While,
Do,
LetMutable,
Match,
With,
When,
As,
Type,
Of,
Module,
Struct,
Sig,
Val,
End,
Direct,
Constraint,
LetHorz,
LetVert,
LetMath,
Controls,
Cycle,
HorzCmdType,
VertCmdType,
MathCmdType,
Command,
Open,
Rec,
Inline,
Block,
Mutable,
Signature,
Include,
Use,
Package,
Math,
Persistent,
LParen,
RParen,
BRecord,
ERecord,
BList,
EList,
BHorzGrp,
EHorzGrp,
BVertGrp,
EVertGrp,
BMathGrp,
EMathGrp,
BPath,
EPath,
ListPunct,
Access,
Arrow,
OverwriteEq,
Bar,
Wildcard,
Colon,
Comma,
Cons,
Coerce,
ExactMinus,
DefEq,
ExactTimes,
ExactAmp,
ExactTilde,
PathCurve,
PathLine,
BinopPlus(String),
BinopMinus(String),
BinopTimes(String),
BinopDivides(String),
BinopEq(String),
BinopLt(String),
BinopGt(String),
BinopAmp(String),
BinopBar(String),
BinopHat(String),
UnopExclam(String),
OptionalType,
OptionalArrow,
Optional,
Omission,
RowVar(String),
HorzCmd(String),
HorzCmdWithMod(Vec<String>, String),
HorzMacro(String),
VertCmd(String),
VertCmdWithMod(Vec<String>, String),
VertMacro(String),
MathCmd(String),
MathCmdWithMod(Vec<String>, String),
VarInHorz(Vec<String>, String),
VarInVert(Vec<String>, String),
VarInMath(Vec<String>, String),
Char(String),
CodeText(String),
Space,
Break,
Item(usize),
Sep,
EndActive,
MathChar(String),
Superscript,
Subscript,
Primes(usize),
Eoi,
}Expand description
One SATySFi token. Variant names and payloads mirror the token declarations
of the v0.0.6 parser.mly (LETNONREC = Let, LAMBDA = Fun, …).
Command payloads keep their sigil, exactly like the OCaml lexer
(HorzCmd("\\emph"), VertCmd("+p")).
The variants listed in the token_leaves! block at the bottom of this file
get a matching leaf struct (KwLet, VarTok, …) with a peek → match →
push-back-on-mismatch Parse/Unparse/Spanned. Multi-field and
multi-variant-matching leaves (LengthTok, LiteralTok, BinOpTok,
VarInHorzTok, …) stay hand-written in crate::leaf.
Variants§
HeaderRequire(String)
HeaderImport(String)
HeaderStage0
HeaderStage1
HeaderPersistent0
Var(String)
VarWithMod(Vec<String>, String)
Constructor(String)
TypeVar(String)
OpenModule(String)
LongUpper(Vec<String>, String)
M.N.P — a dotted module path ending in an UPPER segment
(upstream LONG_UPPER, lexer_v1.mll:357-363). SATySFi 0.1-only:
under 0.0.6 this spelling is a lex error (“module path must end
with a variable name”), unchanged. Payload mirrors
Token::VarWithMod: mods = ["M","N"], final segment "P" —
upstream’s (modidents, modident0) pair carries the same split
(parser_v1.mly:407-413 reassembles the chain from it).
IntConst(i64)
FloatConst(f64)
LengthConst(f64, String)
Literal
Backtick string literal. omit_pre/omit_post mirror the OCaml
LITERAL(_, s, pre, post) flags (whether an adjacent space is trimmed).
Mod
If
Then
Else
Let
LetRec
LetAnd
In
Fun
True
False
Before
While
Do
LetMutable
Match
With
When
As
Type
Of
Module
Struct
Sig
Val
End
Direct
Constraint
LetHorz
LetVert
LetMath
Controls
Cycle
HorzCmdType
VertCmdType
MathCmdType
Command
Open
Rec
rec — SATySFi 0.1-only keyword (val rec/let rec); under 0.0.6
this word stays a plain identifier (see lexer.rs’s version-gated
keyword table).
Inline
inline — SATySFi 0.1-only keyword (val inline \cmd = ...).
Block
block — SATySFi 0.1-only keyword (val block +cmd = ...).
Mutable
mutable — SATySFi 0.1-only keyword (val mutable x <- e/let mutable x <- e in ..); under 0.0.6 this word stays a plain
identifier (see lexer.rs’s version-gated keyword table).
Signature
signature — SATySFi 0.1-only keyword (signature S = sig … end,
upstream lexer_v1.mll:348); a plain identifier under 0.0.6.
Include
include — SATySFi 0.1-only keyword (include M binds /
include S decls, lexer_v1.mll:335); a plain identifier under
0.0.6.
Use
use — SATySFi 0.1-only keyword (Envelopes packaging headers use package … / use … of <path>, saphe-split:parser.mly:371-380 @ b836d512); a plain identifier under 0.0.6 (no 0.0.6 grammar reserves
it — see lexer.rs’s version-gated keyword table).
Package
package — SATySFi 0.1-only keyword (use package …); a plain
identifier under 0.0.6.
Math
math — SATySFi 0.1-only keyword (val math <ctx> \cmd … = …,
parser_v1.mly:452-453 dispatch, MATH reserved at :240); a plain
identifier under 0.0.6 (0.0.6 has no math keyword — the word
survives only as the surface name of BaseType::MathText).
Persistent
persistent — SATySFi 0.1-only keyword, the stage qualifier of a
val persistent ~x = e binding (parser_v1.mly:420-421, decl form
:602-603; reserved at :241, lexed at lexer_v1.mll:345). A plain
identifier under 0.0.6, which spells the same idea per FILE with a
@stage: header rather than per binding.
LParen
RParen
BRecord
ERecord
BList
EList
BHorzGrp
EHorzGrp
BVertGrp
EVertGrp
BMathGrp
EMathGrp
BPath
EPath
ListPunct
Access
Arrow
OverwriteEq
Bar
Wildcard
Colon
Comma
Cons
Coerce
:> — signature coercion/ascription (COERCE, lexer_v1.mll:280).
SATySFi 0.1-only: under 0.0.6 the same two characters lex as
Colon + BinopGt(">"), unchanged.
ExactMinus
DefEq
ExactTimes
ExactAmp
ExactTilde
PathCurve
PathLine
BinopPlus(String)
BinopMinus(String)
BinopTimes(String)
BinopDivides(String)
BinopEq(String)
BinopLt(String)
BinopGt(String)
BinopAmp(String)
BinopBar(String)
BinopHat(String)
UnopExclam(String)
OptionalType
OptionalArrow
Optional
Omission
RowVar(String)
?'r — a SATySFi 0.1 row variable (upstream ROWVAR,
lexer_v1.mll:310-311). RowVarTok
carries the name sans sigil, exactly like Token::TypeVar/
TypeVarTok. V0_1-only: the lexer (lexer.rs’s '?' arm) only ever
mints this under RustyfiVersion::V0_1; under V0_0, ?'r stays
two tokens (OptionalType then TypeVar), byte-identical to before
this addition.
HorzCmd(String)
HorzCmdWithMod(Vec<String>, String)
HorzMacro(String)
VertCmd(String)
VertCmdWithMod(Vec<String>, String)
VertMacro(String)
MathCmd(String)
MathCmdWithMod(Vec<String>, String)
VarInHorz(Vec<String>, String)
VarInVert(Vec<String>, String)
VarInMath(Vec<String>, String)
Char(String)
CodeText(String)
A backtick literal written INSIDE inline text (`…`). Distinct from
Char because upstream keeps it distinct: it reaches the evaluator as
ImInputHorzEmbeddedCodeText and is dispatched to the context’s
code_text_command (evaluator.cppo.ml:768-779), which is how a doc
class sets code spans in a monospace face. Do NOT lex it to a plain
Char run: that erases the distinction irrecoverably, and the literal
then inherits whatever face surrounds it (italic, inside \emph).