Crate lang_forge 

lang_forge

LexerSketch: forge a working language front end — lexer, parser, and lossless syntax tree — from a .lsf schematic.

A schematic is a short NOML document that describes a language: its name, how its tokens look, its grammar, and the capabilities (passes) it includes. Language::from_lsf reads it, checks it, and compiles it into tables; Language::parse then turns source text into a syntax_lang::Node tree with diag_lang::Diagnostics for anything malformed. There is no code generation step and nothing to build: the language is ready the moment the schematic is forged.

lang-forge is the capstone of the -lang language-construction family. Its trees are syntax-lang trees, the family’s lossless CST, which the formatter, incremental reparser, language server, and tree-sitter crates are designed to consume; the adapters that connect a forged language to those crates are not part of lang-forge and arrive with LexerSketch. Its diagnostics render with diag-lang, and its capabilities run on pass-lang.

A first language

use lang_forge::Language;

let calc = Language::from_lsf(
    r##"
    [language]
    name = "calc"

    [lexer]
    line_comments = ["#"]

    [rules]
    program = "stmt*"
    stmt    = "'let' IDENT '=' expr ';' | expr ';'"
    group   = "'(' expr ')'"

    [rules.expr]
    operand = "NUMBER | IDENT | group"
    levels  = [
        { left   = ["+", "-"] },
        { left   = ["*", "/"] },
        { prefix = ["-"] },
        { right  = ["^"] },
    ]
    "##,
)?;

let parse = calc.parse("let area = 3 * r ^ 2; # circle-ish\n");
assert!(!parse.has_errors());

// `^` binds tighter than `*`, so the product's right operand is `r ^ 2`.
let binary = calc.kind("binary").expect("the default operator node");
let product = parse.tree().descendants().find(|n| *n.kind() == binary).expect("3 * r ^ 2");
assert_eq!(product.text(parse.source()), Some("3 * r ^ 2"));
assert_eq!(product.child_nodes().last().and_then(|n| n.text(parse.source())), Some("r ^ 2"));

The rule language

Each entry of [rules] is a rule. A string rule is a sequence of elements:

Element	Matches
'text' or "text"	a keyword (if it looks like an identifier) or a symbol
IDENT, NUMBER, STRING, NEWLINE	a token of that built-in class
name	the rule name, as a child node
a b	a then b
a | b	a, or else b — the first that matches wins
a*, a+, a?	zero or more, one or more, zero or one
( ... )	grouping

A rule builds a node named after itself, unless its name starts with _, in which case its children are placed directly in the parent. The first rule is the start rule unless [language] start names another; its node is the root of every tree.

A table rule, [rules.name], is an expression rule: an operand and operator levels, lowest precedence first. Each level is left, right, or none (binary, by associativity), prefix, or postfix, with an optional then (more grammar after the operator, for calls, indexing, or ?:) and an optional node name.

Format 2

A sketch that says [sketch] format = 2 uses LSF2’s syntax: custom token classes (regular expressions), lexer modes with a mode stack, string classes with interpolation, heredocs, and raw delimiters, contextual and case-insensitive keywords, indentation layout, labelled fields on tree edges, predicates (&e, !e), text back-references, EOF / LINE_START / NL_BEFORE, injections, [ast] supertypes, and a check that rejects greedy repetitions that would silently reject valid input. A sketch with no [sketch] table, or format = 1, forges exactly as lang-forge 1.x did.

use lang_forge::Language;

let lang = Language::from_lsf(
    r##"
    [sketch]
    format = 2

    [language]
    name = "tmpl"
    version = "1.0.0"

    [lexer]
    initial_mode = "text"

    [lexer.tokens]
    OPEN  = { literal = "{{", modes = ["text"], action = "push main" }
    CLOSE = { literal = "}}", action = "pop" }
    VAR   = { regex = '\$[a-z]+' }

    [lexer.modes.text]
    tokens = ["OPEN"]
    text = "TEXT"

    [rules]
    page = "(TEXT | hole)*"
    hole = "OPEN value:VAR filters:('|' IDENT)* CLOSE"
    "##,
)?;

let parse = lang.parse("Hello, {{ $name | upper }}!");
assert!(!parse.has_errors());
let hole = parse.tree().child_nodes().next().expect("a hole");
let fields: Vec<&str> = (0..hole.len())
    .filter_map(|i| lang.field_label(hole, i).and_then(|l| lang.label_name(l)))
    .collect();
assert_eq!(fields, ["value", "filters", "filters"]);

A forged language can be saved as an image (Language::to_image) and loaded without forging (Language::from_image); a sketch can span several files (Sketch); and every diagnostic carries a stable code (LSF for sketches, LF0xxx lexical and LF1xxx parse errors in source).

Errors and recovery

Forging reports every problem in a schematic at once, each with a span into the schematic (Error). Parsing never fails: a missing token is reported and assumed, an unexpected token is reported and wrapped in an ERROR node, and the tree always covers the whole source.

Features

• std (default) — the standard library. Without it the crate is no_std and needs only alloc.

Re-exports

syntax_lang, diag_lang, and pass_lang are re-exported whole, so code that walks trees, renders diagnostics, or writes capability passes names the same versions lang-forge was built against.

Stability

This is 2.0.0-alpha.1, a pre-release of the next major version. The format-1 surface and behaviour of 1.x are kept (format-1 sketches forge and parse exactly as before); the breaking changes are listed in the CHANGELOG with a migration guide. The format-2 surface may still change before 2.0.0. The 1.x promise and what 2.0 changes are set out in docs/API.md.

Re-exports

pub use diag_lang;

pub use pass_lang;

pub use syntax_lang;

Structs

Error

Why a sketch could not be forged into a Language, or a capability pipeline could not be assembled.

Field

A field of a node kind: a label its children carry, the kinds it can hold, and how many (see Language::fields).

Injection

A range of a parsed source covered by an injection (LSF2 §12): another language’s content, or a token of this language whose text has structure of its own. See Parse::injections.

Kind

The kind of a token or node in a forged language’s syntax tree.

Language

A language forged from a .lsf schematic: a lexer, a parser, and the kinds of its syntax tree.

Parse

The result of Language::parse: a lossless syntax tree and the problems found while building it.

Sketch

A sketch made of several files: the entry and the parts its [sketch] modules lists (LSF2 §3.2).

Enums

Cardinality

How many children a field holds (LSF2 §11.3).

ImageError

Why bytes could not be loaded as a language image.

Constants

IMAGE_FORMAT

The image format this lang-forge writes and reads.

Type Aliases

Capability

A capability pass, boxed for Language::pipeline.