Skip to main content

Module parser

Module parser 

Source
Expand description

The parse pipeline: lex → parse_expr (Pratt) + structural (recursive descent) → events → build_tree → rowan CST.

Losslessness is the core invariant: reconstruct(text) == text for all inputs. The grammar is a walking skeleton over a Julia subset and grows incrementally (see TODO.md). Incremental reparse splicing lives in reparse (a single edit) and reparse_edits (a chain of them); both return None when no strategy applies, and the host’s salsa layer (fatou::incremental) then does a full parse.

Structs§

Edit
The edit currency, defined in the sibling edit module and re-exported here so parser::Edit stays the path the parser layer uses. A single contiguous text edit: replace range (a byte range in the old text) with insert.
ParseDiagnostic
A parse-time diagnostic: a classified message anchored to a byte range in the source.
ParseOutput
The result of a parse: the lossless CST plus any diagnostics.
Reparsed
The result of a successful incremental reparse: the new whole-file green tree and its parse diagnostics (absolute offsets in the new text).

Enums§

ReparseTier
Which strategy produced a Reparsed. Surfaced for tests and benchmarks. Ordered cheapest-first, so a chain of reparses can report the most expensive tier it had to reach with Ord::max.
StringDecodeError
Why a STRING_CONTENT token’s source could not be reduced to a value. The two failures matter separately to the s-expression projector, which shows them differently; a consumer that just wants the value treats both as “this literal does not denote a string”.

Constants§

KEYWORDS
Every Julia keyword, as written. Shared by the lexer’s keyword_kind classification and the language server’s keyword completion.

Functions§

apply_edits
The edit currency, defined in the sibling edit module and re-exported here so parser::Edit stays the path the parser layer uses. Apply edits to old left-to-right, each expressed against the text its predecessors produced (the shape LSP didChange batches arrive in). Folds in place, so peak memory is one text rather than one per edit.
diff_edit
The edit currency, defined in the sibling edit module and re-exported here so parser::Edit stays the path the parser layer uses. Recover a single contiguous Edit from a pair of full texts by stripping the common prefix and suffix. Multiple disjoint edits collapse into one spanning edit — still a correct transform, just coarser. Boundaries are clamped to char boundaries of both texts.
is_ident_continue
Whether c may continue an identifier, mirroring JuliaSyntax’s is_identifier_char (Base.is_id_char). ASCII is handled inline; non-ASCII code points defer to the generated unicode_ident tables. The != operator split for a trailing ! is handled by the caller in scan_ident.
is_ident_start
Whether c may begin an identifier, mirroring JuliaSyntax’s is_identifier_start_char (Base.is_id_start_char). ASCII is handled inline; non-ASCII code points defer to the generated unicode_ident tables.
normalize_sexpr
Canonical form of a JuliaSyntax s-expression string. Tokenizes on whitespace and parentheses (preserving "…" string literals as atoms) and rejoins with single-space separation, so pretty-print spacing no longer affects equality.
parse
Parse text into a lossless rowan CST. The drive loop works one logical line at a time: leading trivia (including newlines) is emitted directly at the root, then the line’s statements are parsed with the Pratt parser. A line that carries a top-level ; groups its statements into a TOPLEVEL_SEMICOLON node (mirroring JuliaSyntax’s toplevel-;); a plain line stays bare. An unparseable token is consumed as one error token so the loop always makes progress.
reconstruct
Round-trip the input through the parser: concatenating every token in the CST must reproduce the original text exactly (the losslessness invariant).
reparse
Attempt an incremental reparse of the previous parse (prev_green, parsed from prev_text with prev_diags) under edit, which transforms prev_text into new_text. Returns None when no incremental strategy applies — the caller must then do a full parse.
reparse_edits
Attempt an incremental reparse across a chain of edits, each expressed against the text its predecessors produced — the shape an LSP didChange batch arrives in (see fatou::text::apply_content_changes host-side).
string_value
The value a STRING_CONTENT token’s source text denotes: escapes decoded and line continuations dropped.
to_juliasyntax_sexpr
Render the given Fatou CST as a JuliaSyntax-native s-expression string.
try_apply_edits
The edit currency, defined in the sibling edit module and re-exported here so parser::Edit stays the path the parser layer uses. apply_edits for an edit slice of unproven provenance: None when any edit’s range does not fit the text its predecessors produced.