Struct Language 

pub struct Language { /* private fields */ }

A language forged from a .lsf schematic: a lexer, a parser, and the kinds of its syntax tree.

Forge one with Language::from_lsf (or str::parse), then call parse as often as needed. Forging does all the analysis up front — every rule resolved, every set computed, every conflict refused — so parsing is a walk over precomputed tables that never fails and never panics: malformed input yields a complete tree plus diagnostics.

A Language is immutable once forged. It is Send and Sync, so one language can parse on many threads at once, and Clone when a copy is needed.

The sketch

A sketch (also called a schematic) is a NOML document. A format-1 sketch — the format of lang-forge 1.x, read exactly as 1.x read it — has up to four tables: [language] (the name, and optionally the version, file extensions, and start rule), [lexer] (identifier style, significant newlines, comments, strings), [rules] (the grammar), and [capabilities] (passes the language includes). A sketch that begins with [sketch] format = 2 is read as LSF2, which adds token classes, lexer modes, string classes with interpolation and counted delimiters, keyword policies, layout, field labels, predicates, [ast], and [injections]. The full reference is in docs/API.md; a sketch split over several files is forged from a Sketch.

Examples

use lang_forge::Language;

let calc = Language::from_lsf(
    r##"
    [language]
    name       = "calc"
    version    = "1.0.0"
    extensions = ["calc"]

    [lexer]
    line_comments = ["#"]

    [rules]
    program = "stmt*"
    stmt    = "'let' IDENT '=' expr ';' | expr ';'"

    [rules.expr]
    operand = "NUMBER | IDENT | '(' expr ')'"
    levels  = [
        { left   = ["+", "-"] },
        { left   = ["*", "/"] },
        { prefix = ["-"] },
    ]
    "##,
)?;

let parse = calc.parse("let x = 2 * (3 + 4); # seven, doubled\n-x;");
assert!(!parse.has_errors());

let stmt = calc.kind("stmt").expect("a rule");
assert_eq!(parse.tree().child_nodes().filter(|n| *n.kind() == stmt).count(), 2);

Implementations

impl Language

pub fn to_image(&self) -> Vec<u8>

Writes the forged language as a .lsl image.

Loading the image with from_image gives a language that lexes and parses exactly as this one, without forging the sketch again (Packaged mode, fast startup). The bytes are deterministic: the same sketch forged by the same lang-forge writes the same image on every platform. Forge-time warnings are not part of the image.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[language]\nname = \"sum\"\n[rules]\nsum = \"NUMBER ('+' NUMBER)*\"\n",
)?;
let image = lang.to_image();
assert_eq!(&image[..4], b"LSL\0");

let loaded = Language::from_image(&image).expect("a valid image");
assert_eq!(loaded.parse("1 + 2").dump(), lang.parse("1 + 2").dump());
assert_eq!(loaded.to_image(), image);

pub fn from_image(bytes: &[u8]) -> Result<Language, ImageError>

Loads a language from a .lsl image written by to_image.

The image is treated as untrusted: it is checked completely before it is used (see the module documentation), so any bytes either load as a working language or are refused.

Errors

ImageError::NotAnImage for bytes without the image header, ImageError::Format for an image of another format, ImageError::Corrupt for a truncated or damaged one, and ImageError::Invalid for tables lang-forge does not build.

Examples

use lang_forge::Language;

let lang = Language::from_lsf("[language]\nname = \"n\"\n[rules]\nn = \"NUMBER+\"\n")?;
let loaded = Language::from_image(&lang.to_image()).expect("valid");
assert_eq!(loaded.name(), "n");
assert!(!loaded.parse("1 2 3").has_errors());

impl Language

pub fn from_lsf(schematic: &str) -> Result<Self, Error>

Forges a language from the text of a .lsf schematic.

The schematic is read, checked against the schematic layout, and its grammar compiled and analysed. Everything wrong with it is reported at once.

Errors

Returns an Error carrying one diagnostic per problem, with spans into schematic: NOML syntax errors; unknown, missing, or mistyped settings; malformed rules; undefined rules (with a suggestion); literals the lexer cannot produce; delimiters used twice; left recursion; repetitions of something that can match nothing; alternatives that can never match; and the use of NOML’s dynamic features, which would make the language depend on where it was forged.

Examples

use lang_forge::Language;

let json = Language::from_lsf(
    r#"
    [language]
    name = "json"

    [lexer]
    strings = ['"']

    [rules]
    document = "value"
    value    = "object | array | STRING | NUMBER | 'true' | 'false' | 'null'"
    object   = "'{' (member (',' member)*)? '}'"
    member   = "STRING ':' value"
    array    = "'[' (value (',' value)*)? ']'"
    "#,
)?;
assert!(!json.parse(r#"{"a": [1, true, {"b": null}]}"#).has_errors());
assert!(json.parse(r#"{"a": }"#).has_errors());

A left-recursive rule is refused, with the fix:

use lang_forge::Language;

let err = Language::from_lsf(
    "[language]\nname = \"bad\"\n[rules]\nsum = \"sum '+' NUMBER | NUMBER\"\n",
)
.unwrap_err();
assert_eq!(err.to_string(), "4:1: rule `sum` is left-recursive: sum → sum");

pub fn format(&self) -> u8

The sketch format the language was forged from: 1 or 2.

Examples

use lang_forge::Language;

let v1 = Language::from_lsf("[language]\nname = \"x\"\n[rules]\nx = \"IDENT\"\n")?;
assert_eq!(v1.format(), 1);
let v2 = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"x\"\nversion = \"1.0.0\"\n[rules]\nx = \"IDENT\"\n",
)?;
assert_eq!(v2.format(), 2);

pub fn name(&self) -> &str

The language’s name, from [language] name.

pub fn version(&self) -> Option<&str>

The language’s version, from [language] version, if given.

The text is kept as written; lang-forge does not interpret it.

pub fn extensions(&self) -> impl ExactSizeIterator<Item = &str>

The file extensions of the language’s source files, without the dot, from [language] extensions.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[language]\nname = \"mox\"\nextensions = [\"mox\", \"mx\"]\n[rules]\nfile = \"IDENT*\"\n",
)?;
assert_eq!(lang.extensions().collect::<Vec<_>>(), ["mox", "mx"]);
assert!(lang.extensions().any(|e| e == "mx"));

pub fn capabilities(&self) -> impl ExactSizeIterator<Item = &str>

The capabilities the schematic includes, in the order their passes run, from [capabilities] include.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[language]\nname = \"iron\"\n[rules]\nfile = \"IDENT*\"\n\
     [capabilities]\ninclude = [\"borrow-check\", \"thermal\"]\n",
)?;
assert_eq!(lang.capabilities().collect::<Vec<_>>(), ["borrow-check", "thermal"]);

pub fn kind(&self, name: &str) -> Option<Kind>

The kind called name, or None if the language has no such kind.

Rule names name the nodes rules build (hidden _ rules build none); a keyword or symbol is named by its text; Pratt levels add their node names (binary, prefix, postfix unless renamed); and every language has IDENT, NUMBER, STRING, NEWLINE, WHITESPACE, COMMENT, UNKNOWN, and ERROR. See Kind for the full table.

The lookup is a binary search; look kinds up once and keep them.

Examples

use lang_forge::Language;

let lang = Language::from_lsf("[language]\nname = \"x\"\n[rules]\nitem = \"'go' NUMBER\"\n")?;
assert!(lang.kind("item").is_some());
assert!(lang.kind("go").is_some());
assert!(lang.kind("ERROR").is_some());
assert!(lang.kind("missing").is_none());

pub fn kind_name(&self, kind: Kind) -> &str

The name of kind: the inverse of kind.

A kind is only meaningful to the language that made it. Given a kind from another language, kind_name cannot tell: it returns whatever name this language has at that kind’s position in its kind table — usually a wrong one — or "<unknown>" when this language has fewer kinds than that.

Examples

use lang_forge::Language;

let lang = Language::from_lsf("[language]\nname = \"x\"\n[rules]\nitem = \"'go' NUMBER\"\n")?;
let parse = lang.parse("go 7");
let names: Vec<&str> = parse.tree().tokens().map(|t| lang.kind_name(*t.kind())).collect();
assert_eq!(names, ["go", "WHITESPACE", "NUMBER"]);

// Another language's kinds get a wrong name, or none.
let other = Language::from_lsf(
    "[language]\nname = \"y\"\n[rules]\nlist = \"'[' (pair (',' pair)*)? ']'\"\npair = \"IDENT ':' NUMBER\"\n",
)?;
assert_eq!(lang.kind_name(other.kind("[").expect("a symbol")), "go");
assert_eq!(lang.kind_name(other.kind("pair").expect("a rule")), "<unknown>");

pub fn kind_count(&self) -> usize

How many kinds the language has: valid Kind::index values are 0..kind_count().

Examples

use lang_forge::Language;

let lang = Language::from_lsf("[language]\nname = \"x\"\n[rules]\nitem = \"'go' NUMBER\"\n")?;
let all: Vec<&str> = (0..lang.kind_count() as u16)
    .filter_map(|i| lang.kind_at(i))
    .map(|k| lang.kind_name(k))
    .collect();
assert!(all.contains(&"go") && all.contains(&"item") && all.contains(&"ERROR"));

pub fn kind_at(&self, index: u16) -> Option<Kind>

The kind with index index (see Kind::index), or None if the language has no such kind (ISSUES M04).

The kind comes back with its trivia flag set as this language sets it, so it compares equal to the kinds in this language’s trees. The index of the internal end-of-input marker has no kind.

Examples

use lang_forge::Language;

let lang = Language::from_lsf("[language]\nname = \"x\"\n[rules]\nitem = \"NUMBER\"\n")?;
let space = lang.kind("WHITESPACE").expect("built in");
assert_eq!(lang.kind_at(space.index()), Some(space));
assert_eq!(lang.kind_at(u16::MAX), None);

pub fn root_kind(&self) -> Kind

The kind of every tree’s root: the start rule’s node.

Examples

use lang_forge::Language;

let lang = Language::from_lsf("[language]\nname = \"x\"\n[rules]\nfile = \"NUMBER*\"\n")?;
assert_eq!(lang.root_kind(), lang.kind("file").expect("a rule"));
assert_eq!(*lang.parse("1 2").tree().kind(), lang.root_kind());

pub fn label_name(&self, label: u16) -> Option<&str>

The name of field label label (format 2), or None if the language has no such label.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"x\"\nversion = \"1.0.0\"\n\
     [rules]\nlet_stmt = \"'let' name:IDENT '=' value:NUMBER\"\n",
)?;
let name = lang.label_id("name").expect("a label");
assert_eq!(lang.label_name(name), Some("name"));
assert_eq!(lang.label_id("missing"), None);

pub fn label_id(&self, name: &str) -> Option<u16>

The id of the field label called name, the number lower-lang’s Pick::Label takes. Labels are numbered by first occurrence over the rules in order (LSF2 §5.4).

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"x\"\nversion = \"1.0.0\"\n\
     [rules]\nassign = \"target:IDENT '=' value:NUMBER\"\n",
)?;
assert_eq!(lang.label_id("target"), Some(0));
assert_eq!(lang.label_id("value"), Some(1));
assert_eq!(lang.label_id("missing"), None);

pub fn field_label(&self, parent: &Node<Kind>, index: usize) -> Option<u16>

The field label of parent’s child at index (counting every child, trivia included, as Node::children yields them), or None for an unlabelled child or an index out of range.

This has the shape of lower-lang’s Labeler::label, so the adapter that hands a format-2 tree’s labels to Lowerer::with_labeler is one line: fn label(&self, p: &Node<Kind>, i: usize) -> Option<u16> { self.0.field_label(p, i) }. Labels live on the tree itself (each child’s kind carries the label of its edge), so this works on any tree the language built, cloned or not.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"w\"\nversion = \"1.0.0\"\n\
     [rules]\nwhile_stmt = \"'while' cond:IDENT body:block\"\nblock = \"'{' '}'\"\n",
)?;
let parse = lang.parse("while ready { }");
let root = parse.tree();
let names: Vec<Option<&str>> = (0..root.len())
    .map(|i| lang.field_label(root, i).and_then(|l| lang.label_name(l)))
    .collect();
assert_eq!(names, [None, None, Some("cond"), None, Some("body")]);

pub fn fields(&self, kind: Kind) -> impl Iterator<Item = Field<'_>>

The fields of node kind kind (format 2): every label its children can carry, with the kinds the field can hold and its cardinality, derived from the grammar (LSF2 §11.3). Empty for a kind with no labelled children, and for every kind of a format-1 language.

Examples

use lang_forge::{Cardinality, Language};

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"c\"\nversion = \"1.0.0\"\n\
     [rules]\ncall = \"callee:IDENT '(' (args:NUMBER (',' args:NUMBER)*)? ')' tail:';'?\"\n",
)?;
let call = lang.kind("call").expect("a rule");
let fields: Vec<(&str, Cardinality)> =
    lang.fields(call).map(|f| (f.name(), f.cardinality())).collect();
assert_eq!(
    fields,
    [("callee", Cardinality::One), ("args", Cardinality::Many), ("tail", Cardinality::Optional)]
);

pub fn supertype( &self, name: &str, ) -> Option<impl ExactSizeIterator<Item = Kind> + '_>

The members of [ast] supertype name (format 2), with supertypes of supertypes expanded, or None if there is no such supertype.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"s\"\nversion = \"1.0.0\"\n\
     [rules]\nfile = \"(num | word)*\"\nnum = \"NUMBER\"\nword = \"IDENT\"\n\
     [ast]\nAtom = [\"num\", \"word\"]\n",
)?;
let atoms: Vec<&str> = lang.supertype("Atom").expect("declared").map(|k| lang.kind_name(k)).collect();
assert_eq!(atoms, ["num", "word"]);
assert_eq!(lang.supertypes().collect::<Vec<_>>(), ["Atom"]);

pub fn supertypes(&self) -> impl Iterator<Item = &str>

The names of the [ast] supertypes, in sketch order.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"s\"\nversion = \"1.0.0\"\n\
     [rules]\nfile = \"(num | word)*\"\nnum = \"NUMBER\"\nword = \"IDENT\"\n\
     [ast]\nLiteral = [\"num\"]\nAtom = [\"Literal\", \"word\"]\n",
)?;
assert_eq!(lang.supertypes().collect::<Vec<_>>(), ["Literal", "Atom"]);

pub fn warnings(&self) -> &[Diagnostic]

Warnings found while forging (format 2): checks set to warn, such as unused rules (LSF4302) and unused token classes (LSF3401), and keys LSF2 specifies that this release does not check. Spans are into the sketch, as for Error.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"w\"\nversion = \"1.0.0\"\n\
     [rules]\nfile = \"IDENT*\"\nforgotten = \"NUMBER\"\n",
)?;
let warning = &lang.warnings()[0];
assert_eq!(warning.code().map(|c| c.to_string()).as_deref(), Some("LSF4302"));
assert_eq!(warning.message(), "rule `forgotten` is never used");

pub fn display_name(&self) -> &str

The language’s display name, from [language] display_name (format 2), or its name.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"mox\"\nversion = \"0.1.0\"\n\
     display_name = \"Mox\"\ndescription = \"A modern PHP.\"\nedition = \"2026\"\n\
     [rules]\nfile = \"IDENT*\"\n",
)?;
assert_eq!(lang.display_name(), "Mox");
assert_eq!(lang.description(), Some("A modern PHP."));
assert_eq!(lang.edition(), Some("2026"));

let plain = Language::from_lsf("[language]\nname = \"p\"\n[rules]\nfile = \"IDENT*\"\n")?;
assert_eq!((plain.display_name(), plain.description(), plain.edition()), ("p", None, None));

pub fn description(&self) -> Option<&str>

[language] description (format 2), if given (see display_name for an example).

pub fn edition(&self) -> Option<&str>

[language] edition (format 2), if given (see display_name for an example).

pub fn shebang_names(&self) -> impl Iterator<Item = &str>

[language] shebang_names (format 2): the interpreter names that identify the language in a #! line, for editors.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"m\"\nversion = \"0.1.0\"\n\
     shebang_names = [\"m\", \"mscript\"]\n[rules]\nfile = \"IDENT*\"\n",
)?;
assert_eq!(lang.shebang_names().collect::<Vec<_>>(), ["m", "mscript"]);

pub fn lex(&self, source: &str) -> Vec<Token<Kind>>

Splits source into tokens, trivia included.

The tokens are contiguous and cover the whole source, so this is the stream a syntax highlighter wants. Characters that begin no token come back as UNKNOWN tokens; parse reports them, lex does not. A source of 4 GiB or more, which spans cannot address, yields no tokens.

Examples

use lang_forge::Language;
use lang_forge::syntax_lang::TokenKind;

let lang = Language::from_lsf(
    "[language]\nname = \"x\"\n[lexer]\nline_comments = [\"--\"]\n[rules]\nfile = \"IDENT*\"\n",
)?;
let tokens = lang.lex("alpha -- note\nbeta");
let significant: Vec<&str> = tokens
    .iter()
    .filter(|t| !t.is_trivia())
    .map(|t| lang.kind_name(*t.kind()))
    .collect();
assert_eq!(significant, ["IDENT", "IDENT"]);
assert_eq!(tokens.len(), 5); // IDENT, WHITESPACE, COMMENT, WHITESPACE, IDENT

pub fn parse<'a>(&'a self, source: &'a str) -> Parse<'a>

Parses source into a lossless syntax tree.

Never fails: problems become diagnostics on the returned Parse and the tree is complete regardless, with unexpected tokens wrapped in ERROR nodes. Input nested too deeply to parse is reported rather than followed: the parser recurses at most 768 grammar levels, which needs at most about 256 KiB of stack in a release build (768 KiB in a debug build) and allows well over a hundred levels of nesting in typical grammars.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[language]\nname = \"block\"\n[rules]\nblock = \"'{' stmt* '}'\"\nstmt = \"IDENT ';'\"\n",
)?;

let good = lang.parse("{ a; b; }");
assert!(!good.has_errors());

// A stray `;` is skipped, the rest still parses.
let bad = lang.parse("{ a; ; b; }");
assert_eq!(bad.diagnostics().len(), 1);
assert_eq!(bad.diagnostics()[0].message(), "expected stmt, found `;`");
let error = lang.kind("ERROR").expect("built in");
assert_eq!(bad.tree().descendants().filter(|n| *n.kind() == error).count(), 1);

pub fn parse_file<'a>(&'a self, extension: &str, source: &'a str) -> Parse<'a>

Parses source as a file with extension: with the lexer mode and start rule [language] files gives that extension (format 2), or as parse does when it gives none.

Examples

use lang_forge::Language;

let lang = Language::from_lsf(
    "[sketch]\nformat = 2\n[language]\nname = \"x\"\nversion = \"1.0.0\"\n\
     extensions = [\"x\", \"xs\"]\nfiles = { xs = { start = \"item\" } }\n\
     [rules]\nfile = \"item*\"\nitem = \"NUMBER\"\n",
)?;
assert_eq!(lang.parse_file("xs", "7").tree().kind(), &lang.kind("item").expect("a rule"));
assert_eq!(lang.parse_file("x", "7 8").tree().kind(), &lang.kind("file").expect("a rule"));

pub fn pipeline<'a>( &self, passes: impl IntoIterator<Item = Capability>, ) -> Result<PassManager<Parse<'a>>, Error>

Assembles the language’s capability pipeline from a registry of passes.

passes may hold passes for many languages; the pipeline takes the ones whose Pass::name the schematic’s [capabilities] include lists, in that order, and ignores the rest. Run it over each Parse with PassManager::run.

Errors

Returns an Error with a diagnostic, pointing into the schematic, for every included capability that has no pass in passes or more than one.

Examples

use lang_forge::diag_lang::{Diagnostic, Label, Severity};
use lang_forge::pass_lang::{Outcome, Pass, PassError};
use lang_forge::{Capability, Language, Parse};

/// Warns about every identifier written in capitals.
struct Shouting;

impl<'a> Pass<Parse<'a>> for Shouting {
    fn name(&self) -> &'static str {
        "no-shouting"
    }

    fn run(&mut self, parse: &mut Parse<'a>) -> Result<Outcome, PassError> {
        let ident = parse.language().kind("IDENT").ok_or_else(|| PassError::new("no IDENT"))?;
        let loud: Vec<_> = parse
            .tree()
            .tokens()
            .filter(|t| *t.kind() == ident)
            .filter(|t| {
                let text = &parse.source()[t.span().start().to_usize()..t.span().end().to_usize()];
                text.len() > 1 && text.chars().all(|c| c.is_ascii_uppercase())
            })
            .map(|t| t.span())
            .collect();
        for span in loud {
            parse.report(Diagnostic::new(Severity::Warning, "no need to shout", Label::unlabelled(span)));
        }
        Ok(Outcome::Unchanged)
    }
}

let lang = Language::from_lsf(
    "[language]\nname = \"words\"\n[rules]\nfile = \"IDENT*\"\n\
     [capabilities]\ninclude = [\"no-shouting\"]\n",
)?;
let registry: Vec<Capability> = vec![Box::new(Shouting)];
let mut pipeline = lang.pipeline(registry)?;

let mut parse = lang.parse("quiet LOUD calm");
pipeline.run(&mut parse).expect("the pass succeeds");
assert_eq!(parse.diagnostics().len(), 1);
assert_eq!(parse.diagnostics()[0].message(), "no need to shout");

impl Language

pub fn from_sketch(sketch: &Sketch) -> Result<Self, Error>

Forges a language from a Sketch of one or more files (LSF2 §3).

The first file is the entry. A format-2 entry lists its parts in [sketch] modules, and every part must be in the sketch with [sketch] kind = "part". A one-file format-1 sketch is forged exactly as from_lsf forges it.

Errors

Returns an Error whose diagnostics point into the sketch’s source_map: everything from_lsf reports, plus a module that is not in the sketch or listed twice (LSF2023, LSF2005), a file no module lists (LSF2023), a part that is not marked kind = "part" or has [language] or modules (LSF2024, LSF2022, LSF2003), a key defined in two files (LSF2002), and a missing [language] start when rules span files (LSF2006).

Examples

use lang_forge::{Language, Sketch};

let mut sketch = Sketch::new();
sketch.add("words.lsf", "[sketch]\nformat = 2\nmodules = [\"lexer.lsf\"]\n\
    [language]\nname = \"words\"\nversion = \"1.0.0\"\n[rules]\nfile = \"TAG*\"\n")?;
sketch.add("lexer.lsf", "[sketch]\nformat = 2\nkind = \"part\"\n\
    [lexer.tokens]\nTAG = { regex = \"#[a-z]+\" }\n")?;
assert_eq!(sketch.len(), 2);

let words = Language::from_sketch(&sketch)?;
assert!(!words.parse("#a #bc").has_errors());

Trait Implementations

impl Clone for Language

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Language

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromStr for Language

fn from_str(schematic: &str) -> Result<Self, Error>

Forges a language; the same as Language::from_lsf.

use lang_forge::Language;

let lang: Language = "[language]\nname = \"n\"\n[rules]\nn = \"NUMBER\"\n".parse()?;
assert_eq!(lang.name(), "n");

type Err = Error

The associated error which can be returned from parsing.