meta-language (Rust)

A Rust foundation for a universal, self-describing meta language backed by a links network. The initial crate focuses on the common structural substrate: links, references, source spans, parse status metadata, configurable trivia attachment, self-description roots, and verification that a parsed region is clean.

Website: https://link-foundation.github.io/meta-language — project description, an interactive WebAssembly demo, and the full Rust API documentation.

What Is Implemented

• A mutable LinkNetwork where every item is a link.
• Self-referential point links, so a point is represented without introducing a separate primitive.
• Relation links as ordinary links with ordered references to other links.
• Field labels as explicit links instead of side-table metadata.
• Source metadata: link type, named/anonymous flag, byte range, row/column points, and is_error, has_error, is_missing, is_extra flags.
• verify_full_match() for reporting error and missing links in a selected source region.
• parse() as the default lossless parse entry point; the explicit parse_lossless_text() boundary remains available.
• reconstruct_text() for byte-for-byte reconstruction from non-missing token links ordered by source span.
• insert_source_token(), insert_syntax_node(), and render_source() for emitting target-language source from programmatically constructed syntax networks whose token leaves do not come from a prior parse.
• projected_links() for viewing the same lossless network as concrete syntax, abstract syntax, or semantic-only data by stripping lower-level preservation links from the view.
• NetworkSnapshot and MutableNetworkSnapshot for immutable versioned snapshots, editable forks, provenance, and forward commits.
• AccessMode for a read-only or mutable engine per user configuration: freeze() / as_read_only() yield a ReadOnlyNetwork view whose mutators are unreachable at compile time, and parse_engine() returns an EngineNetwork that rejects mutation with a clear diagnostic under AccessMode::ReadOnly; the frozen form reuses snapshot Arc sharing.
• LinkStore and EngineLinkStore for storage-backed create/read/update/delete and search operations: reads take &self, writes take &mut self, the default store is the in-memory LinkNetwork, and read-only access mode rejects writes through the same storage boundary.
• Optional doublets feature support for a file-mapped DoubletsLinkStore using doublets-rs 0.4, with lossless network round trips and a documented doublets-web backend label for browser/WASM exchange of the same binary graph layout.
• ParseConfiguration with containment-link, token-link, or combined trivia attachment policies.
• Mixed-region links for Markdown fenced code and HTML regions, plus HTML script, style, and style-attribute regions, with txt fallback for prose regions that content sniffing cannot classify.
• LinkQuery for structural matching by link type, term, language, named flag, tree-sitter-query-like S-expressions, captures, and host predicates.
• QueryPlan for versioned engine-neutral SELECT, INSERT, UPDATE, and DELETE semantics, with registry-driven GraphQL and SQL adapters, common aggregates and query clauses, vendor-profile normalization, and CST-backed semantic provenance.
• find() / replace() for codemod-style query transforms over captured links while preserving unchanged source bytes.
• SubstitutionRule / apply_substitution() for the link-cli-style match-and-substitute operation.
• apply_edit() for incremental source reparsing with stable outside-edit link ids, snapshot fork sharing for unchanged links, and structural diff sets for changed, added, and removed links.
• Concept-to-language syntax mappings for cross-language reconstruction.
• reconstruct_text_as() for semantic cross-language reconstruction and configurable formalization levels.
• Recursive TranslationRuleSet evaluation with named reference captures, variadic and optional template segments, target sub-language fallbacks, and indentation-aware substitution. See docs/translation-rules.md.
• Exact-match concept interning with language-bound expression links, queryable external-id aliases, LiNo concept-set import, and seed_common_concept_ontology() for the default 351-concept semantic lexicon plus structural programming-language concepts.
• Concept records for every canonical concept and operation, shared with the JavaScript runtime (concept_records(), seed_concept_records(), import_concept_ontology()). Each record has a readable English phrase, a definition, constraints, source aliases and former names.
• Object-identity links, many-valued TruthValue semantics, and fixed-point ProbabilisticTruthValue confidence semantics.
• A testable parity registry and upstream-provenanced PARITY_FIXTURES for executable competitor and ecosystem feature gates.
• A language-neutral executable query-plan IR plus a fail-closed, registry-driven GraphQL adapter for queries, mutations, filters, ordering, pagination, grouping, common aggregates, and source/CST provenance. See docs/query-plans.md.
• Structural LiNo parsing for links-notation doublets, triplets, named links, simple indented definitions, and self-references.
• LANGUAGE_FIXTURES with lossless parse/reconstruction samples for every required markup, programming-language, and natural-language target.
• NATURAL_LANGUAGE_GRAMMAR_FIXTURES with pass/fail grammaticality fixtures for the ten natural-language targets, including provenance for the repo-authored sentences and UD-derived tag vocabulary.
• Coverage targets for full txt, Markdown, HTML, PDF, and DOCX support, mixed grammar embedding, ten programming-language parser targets, and ten natural-language parser targets.
• A documented text PDF profile (issue #84): render_pdf_document() / parse_pdf_document() map a language-free formatting document onto a valid, uncompressed single-page PDF (marked content for heading/paragraph/list, font resources for bold/italic), parse("…", "pdf", …) builds a byte-exact lossless network with additive concept-tagged structure links, and reconstruct_text_as("PDF", …) renders a structurally equivalent PDF through the shared concept layer. See docs/pdf-fidelity.md for the round-trip fidelity matrix.
• A documented DOCX (OOXML) profile (issue #85): render_docx_document() / parse_docx_document() map a language-free formatting document onto WordprocessingML word/document.xml (<w:pStyle> headings, <w:numPr> lists, <w:b/>/<w:i/> bold/italic runs), a binary OPC layer (render_docx_package() / parse_docx_package()) assembles a valid .docx ZIP with no new dependencies, parse("…", "docx", …) builds a byte-exact lossless network with additive concept-tagged structure links, and reconstruct_text_as("DOCX", …) renders a structurally equivalent DOCX through the shared concept layer. See docs/docx-fidelity.md for the round-trip fidelity matrix.
• Cross-format document reconstruction (issue #86): reconstruct_text_as("txt" | "Markdown" | "HTML" | "PDF" | "DOCX", …) reconstructs a document parsed from any supported format into any other over the shared concept layer (same-format targets stay byte-exact), with txt added as a first-class format and the lossy fallback floor. Per-format capability profiles (document_format_profile, DOCUMENT_FORMATS, CROSS_FORMAT_CONCEPTS) report, for every concept, either native support or a documented LanguageProfile fallback. See docs/cross-format-fidelity.md for the cross-format entry point and the per-format fidelity matrix.
• Self-description roots for link, reference, relation link, language, grammar, type, Type, concept, point, field, trivia, region, and object.
• A lossless text parser boundary that preserves tokens, trivia, recovery markers, and mixed-region metadata behind the same representation.
• Versioned JavaScript, Rust, Lean, and Rocq/Coq parser/emitter capability reports, a shared conformance corpus with the JavaScript package, and all 12 directed translation contracts failing closed until semantic preservation exists. See docs/four-language-contracts.md and the full delivery checklist in docs/issue-195-requirement-ledger.md.

Usage

use meta_language::{LinkNetwork, ParseConfiguration};

let network = LinkNetwork::parse("alpha beta", "plain-text", ParseConfiguration::default());
let report = network.verify_full_match(None);

assert!(report.is_clean());
assert_eq!(network.reconstruct_text(), "alpha beta");

The repository also includes a native ESM JavaScript package in js/:

cd js
npm ci
npm test

import { LinkNetwork, ParseConfiguration } from 'meta-language';

const network = LinkNetwork.parse('alpha beta', 'txt', ParseConfiguration.default());

console.log(network.reconstructText());

The JavaScript package is checked against the Rust feature surface through parity/language-features.json and npm run check:parity.

The default parse path is lossless. Callers that need a narrower view can use a projection without mutating the original network:

use meta_language::{LinkNetwork, NetworkProjection, ParseConfiguration};

let network = LinkNetwork::parse("alpha beta", "plain-text", ParseConfiguration::default());
let abstract_links = network
    .projected_links(NetworkProjection::AbstractSyntax)
    .count();

assert!(abstract_links < network.len());

Construct source directly as a syntax network when code should be generated before validation:

use meta_language::{LinkNetwork, ParseConfiguration};

let mut network = LinkNetwork::new();
let tokens = [
    network.insert_source_token("JavaScript", "const answer = "),
    network.insert_source_token("JavaScript", "42"),
    network.insert_source_token("JavaScript", ";\n"),
];
let declaration = network.insert_syntax_node("JavaScript", "lexical_declaration", tokens);
network.insert_syntax_node("JavaScript", "program", [declaration]);

let source = network.render_source("JavaScript");
assert_eq!(source, "const answer = 42;\n");
assert!(LinkNetwork::parse(&source, "JavaScript", ParseConfiguration::default())
    .verify_full_match(None)
    .is_clean());

Configure the engine read-only when a parsed network must never be mutated. The frozen view exposes every read operation but no mutators (calling one is a compile error), and the EngineNetwork boundary rejects mutation at runtime:

use meta_language::{AccessMode, LinkNetwork, ParseConfiguration};

let configuration = ParseConfiguration::default().with_access_mode(AccessMode::ReadOnly);
let mut engine = LinkNetwork::parse_engine("alpha beta", "plain-text", configuration);

assert!(engine.is_read_only());
assert_eq!(engine.reconstruct_text(), "alpha beta");
assert!(engine.as_mutable().is_err()); // read-only engine rejects mutation

Use the storage trait directly when links need to move between in-memory and binary stores. The optional doublets backend is enabled with --features doublets:

use meta_language::{
    DoubletsLinkStore, LinkNetwork, LinkStore, LinkStoreQuery, ParseConfiguration,
};

let network = LinkNetwork::parse("const answer = 42;\n", "JavaScript", ParseConfiguration::default());
let mut store = DoubletsLinkStore::create_file("network.doublets").expect("create doublets store");
store.replace_with_network(&network).expect("write network");

let restored = DoubletsLinkStore::open_file("network.doublets")
    .expect("open doublets store")
    .to_network()
    .expect("read network");
assert_eq!(restored.to_lino(), network.to_lino());

let links = LinkStore::search(&restored, &LinkStoreQuery::new()).expect("search links");
assert_eq!(links.len(), restored.len());

LinkStoreBackend::DoubletsWeb names the WASM/browser exchange target for this binary graph representation; native code uses DoubletsLinkStore, while a browser host can map the same logical records through doublets-web.

Codemod-style transforms can select links with an S-expression query and replace only captured source ranges:

use meta_language::{LinkNetwork, LinkQuery, ParseConfiguration, ReplacementRule};

let mut network = LinkNetwork::parse(
    "const oldName = call(oldName);\n",
    "JavaScript",
    ParseConfiguration::default(),
);
let query = LinkQuery::from_sexpression(
    r#"
    (identifier) @target
    (#eq? @target "oldName")
    "#,
)
.expect("query parses");
let captures = network.find(&query);

network.replace(
    &captures,
    &ReplacementRule::captured_text("target", "newName"),
);

assert_eq!(network.reconstruct_text(), "const newName = call(newName);\n");

Cross-language reconstruction can naturalize a parsed semantic proposition into another target language, or expose progressively more formal representations:

use meta_language::{FormalizationLevel, LinkNetwork, ParseConfiguration};

let network = LinkNetwork::parse(
    "Hawaii is a state.\n",
    "English",
    ParseConfiguration::default(),
);

assert_eq!(
    network.reconstruct_text_as("Russian", ParseConfiguration::default()),
    "Гавайи это штат.\n"
);
assert_eq!(
    network.reconstruct_text_as(
        "Russian",
        ParseConfiguration::default().with_formalization_level(FormalizationLevel::Concept),
    ),
    "statehood(Q782, Q35657)\n"
);

The same reconstruct_text_as entry point translates a document parsed from one markup/document format into another over the shared formatting concept layer. Same-format targets stay byte-exact; cross-format targets carry the document's heading/paragraph/list and bold/italic/link structure, degrading through the documented per-format fallbacks (see docs/cross-format-fidelity.md):

use meta_language::{LinkNetwork, ParseConfiguration};

let network = LinkNetwork::parse(
    "# Status Report\n\nThe system is **ready** for *launch*.",
    "Markdown",
    ParseConfiguration::default(),
);

let html = network.reconstruct_text_as("HTML", ParseConfiguration::default());
assert!(html.contains("<h1>Status Report</h1>"));
assert!(html.contains("<strong>ready</strong>"));

// txt is the lossy fallback floor: prose survives, markup is dropped.
let txt = network.reconstruct_text_as("txt", ParseConfiguration::default());
assert!(txt.contains("The system is ready for launch"));
assert!(!txt.contains('#'));

CLI

cargo run -- describe
cargo run -- verify --language plain-text --text "alpha beta"
cargo run -- infer examples/json-samples/ --format bnf --metrics
cargo run -- import-grammar --format bnf tests/fixtures/grammar/bnf/arithmetic.bnf
cargo run -- emit-grammar --format gbnf grammar.lino --out grammar.gbnf
cargo run -- translate-grammar grammar.lino --from-language en --to-language ru

describe prints the built-in self-description network as LiNo-style definition lines that round-trip through parse() and reconstruct_text(). verify parses the text with the lossless text boundary and exits successfully when the resulting region has no error or missing links.

The grammar subcommands expose the grammar IR pipeline from the shell. infer reads one or more example files or directories, infers a grammar, and writes LiNo by default; --format bnf|ebnf|abnf|peg|gbnf|tree-sitter renders another supported notation, --metrics writes precision/recall/F1/runtime to stderr, and --out <path> writes the grammar to a file. import-grammar reads bnf|ebnf|abnf|peg|antlr|lark|gbnf|tree-sitter input and renders LiNo by default or another --to notation. emit-grammar reads LiNo or native grammar surface input and emits bnf|ebnf|abnf|peg|gbnf|tree-sitter. Unsupported target formats report a clear non-zero CLI error. translate-grammar rewrites concept-aligned rule names and documentation to the requested target language while preserving grammar structure.

The grammar subcommand is the grammar interchange tool shared with the JavaScript package (npx meta-language grammar ...). It prints the same output and exits with the same status in both runtimes:

cargo run -- grammar formats
cargo run -- grammar import --from abnf message.abnf
cargo run -- grammar validate --from native grammar.grammar
cargo run -- grammar convert --from pest --to ebnf message.pest
cargo run -- grammar export --to abnf grammar.grammar
cargo run -- grammar merge --source bnf:message.bnf --source ebnf:message.ebnf
cargo run -- grammar rename --from ebnf --rule word --name name message.ebnf
cargo run -- grammar round-trip --from abnf --accept "1+2" --reject "1+x" sum.abnf

It reads and writes abnf, antlr, bnf, ebnf, gbnf, lark, pest, tree-sitter-json and native, a line-per-rule listing of the grammar model that import prints and export reads. Lossy conversion steps, merge decisions and rename aliases go to standard error. The exit status is 0 on success, 1 when a problem was found (a validation error, an unresolved required merge equivalence or a broken round trip) and 2 on a usage or input error. grammar help lists every option.

check_grammar_reverse_conversion checks the cycle source grammar -> native links -> exported grammar -> native links. The emitter only sees the grammar read back from render_grammar_links, and the re-imported export must keep every rule, kind, definition and documentation comment and give the same answers on the accept and reject samples. For an exact copy of the source, import_grammar_lossless also returns a GrammarLayout (definition texts, comments and spacing) that render_grammar_layout_links writes as links. emit_grammar_lossless then rebuilds the source byte for byte and writes only changed or new rules fresh.

lower_grammar writes a grammar in a notation that cannot express all of it (abnf, antlr, bnf, ebnf, gbnf, lark, pest or tree-sitter-json). Each construct the target cannot write moves into a helper rule encoded in constructs it can write. For example, an optional item becomes a choice with the empty expression and a repetition becomes a recursive rule. The result has two parts: the executable text, which the target's own importer reads back, and reconstruction metadata as links. The metadata names every helper, its construct, whether its encoding is exact or approximate and the original expression, plus every rename, kind and documentation the target does not keep. reconstruct_grammar rebuilds the original from both parts. check_grammar_lowering reports every emission note, every executable that does not read back, every feature the reconstruction lost and every sample an exact lowering disagrees on.

translate_program(source, from, to) carries every Lean and Rocq theorem as a TranslationObligation in semantics().obligations. The obligation names the source theorem and the target declaration that states it, so a false restatement is rejected by the source kernel and by every target. Into Lean and Rocq the obligation is discharged by the target kernel itself (discharge is target-kernel). The emitted proof has no sorry, admit or axiom, and #print axioms or Print Assumptions shows it closed without an axiom of the translation or of another kernel. Into JavaScript and Rust a theorem is a bounded check (discharge is source-kernel and check is bounded), not a proof. Run with --ml-check-theorems, the program prints theorem <name>: holds on the bounded domain or fails, and the proof stays with the source kernel. A statement that holds on the bounded domain but not in general therefore passes the check while every kernel rejects it.

Grammar subsystem

The grammar layer stores authored, imported, inferred, translated, and generated grammars as first-class links instead of side data. Its IR is built from Grammar, GrammarRule, GrammarExpr, RuleKind, and GrammarFormat, lowers through ToLinks / FromLinks as LinkType::Grammar, and is documented in docs/grammar.

Start with the architecture overview for the IR, links encoding, and pipeline. The stage guides cover authoring, import and export, code generation, inference, translation, and CLI/runtime integration.

Parity Implementation

The crate exposes PARITY_TARGETS, MARKUP_LANGUAGE_TARGETS, PROGRAMMING_LANGUAGE_TARGETS, SECOND_TIER_PROGRAMMING_LANGUAGE_TARGETS, NATURAL_LANGUAGE_TARGETS, DATA_FORMAT_TARGETS, and GRAMMAR_EMBEDDING_TARGETS so comparison scope is part of the tested Rust API. It also exposes PARITY_FIXTURES, with executable, provenance-tracked fixtures covering every advertised target capability, and LANGUAGE_FIXTURES, with a lossless fixture for every requested language target. Grammar notation round-trip fidelity is tracked in docs/grammar/fidelity.md through the grammar_format_profile API. The current registry tracks tree-sitter, LibCST, Recast, jscodeshift, Rowan, cstree, Roslyn, links-notation, link-cli, lino-objects-codec, relative-meta-logic, formal-ai, and meta-expression. Internal ecosystem fixtures now structurally parse links-notation doublet, triplet, tuple, indented, and self-reference cases; link-cli create/update/delete/swap substitutions; lino object round-trip, shared-reference, and circular-reference cases; relative-meta-logic dependent, many-valued, probabilistic, and liar-paradox cases; formal-ai seed and benchmark .lino corpora; and meta-expression formalize and naturalize examples backed by the verified 351-concept lexicon.

See docs/parity-roadmap.md for the feature matrix, executable fixture gates, and language coverage targets.

Website

The project website at https://link-foundation.github.io/meta-language is built and deployed by the Deploy Website job in .github/workflows/rust.yml. It is assembled by scripts/build-site.rs into _site/:

Build and preview the site locally:

# Requires the wasm32-unknown-unknown target and wasm-pack:
#   rustup target add wasm32-unknown-unknown
#   cargo install wasm-pack
rust-script scripts/build-site.rs
python3 -m http.server -d _site 8000   # open http://localhost:8000

Deploying cargo doc output directly used to make the Pages root URL return HTTP 404 because cargo doc writes target/doc/<crate>/index.html but no root index.html (see issue #90 and docs/case-studies/issue-90). The assembled site puts the landing page at the root and keeps the API docs under /api/.

One-time setup: in the repository's Settings → Pages, set Source to GitHub Actions. Without this, the first deploy fails on actions/deploy-pages with "Get Pages site failed". This cannot be configured from a workflow.

Development

cargo fmt --check
cargo clippy --all-targets --all-features
cargo test --all-features
rust-script scripts/check-no-src-tests.rs

The WebAssembly demo crate in web/ is standalone (not part of the published crate). Check it with:

cd web && cargo test && cargo clippy --all-targets

This repository uses changelog fragments in changelog.d/; code changes should include a fragment with the intended semantic-version bump.