textmate-lang 0.2.0

The model

• A Grammar is a scope name (source.demo), top-level patterns, a repository of named rules, and injections. to_json validates it and returns the JSON.
• A Rule is one entry of a pattern list: a Match, a BeginEnd or BeginWhile region, an include of a repository entry, $self, $base, or another grammar, or a list of rules. A Capture says what to do with a capture group.
• An Error says what is wrong and where, as a JSON Pointer into the output (/repository/strings/end); a regular-expression error also gives the byte offset in the pattern.
• validate_regex and escape_regex check one pattern, and turn literal text (keywords, operators) into one.

What it guarantees, and how each guarantee is checked:

Installation

[dependencies]
textmate-lang = "0.2"

Without the standard library:

[dependencies]
textmate-lang = { version = "0.2", default-features = false }

Quick start

use textmate_lang::{BeginEnd, Grammar, Match, Rule};

let json = Grammar::new("source.demo")
    .name("Demo")
    .file_type("demo")
    .pattern(Rule::include("comments"))
    .pattern(Rule::include("strings"))
    .pattern(Match::new(r"\b(if|else|while|return)\b").scope("keyword.control.demo"))
    .pattern(
        Match::new(r"\b(fn)\s+([a-z_]\w*)")
            .capture(1, "storage.type.function.demo")
            .capture(2, "entity.name.function.demo"),
    )
    .repository("comments", Match::new("#.*$").scope("comment.line.number-sign.demo"))
    .repository(
        "strings",
        BeginEnd::new("\"", "\"")
            .scope("string.quoted.double.demo")
            .pattern(Match::new(r"\\.").scope("constant.character.escape.demo")),
    )
    .to_json()?;

assert!(json.contains("\"scopeName\": \"source.demo\""));
// std::fs::write("syntaxes/demo.tmLanguage.json", json)?;
# Ok::<(), textmate_lang::Error>(())

Mistakes caught before an editor sees them

use textmate_lang::{BeginEnd, Error, Grammar, Match, RegexError, RegexErrorKind, Rule};

// A pattern outside the subset: `x{2}?` is lazy in some engines and
// optional in Oniguruma.
let err = Grammar::new("source.demo").pattern(Match::new("x{2}?")).validate().unwrap_err();
assert_eq!(
    err,
    Error::InvalidRegex {
        path: "/patterns/0/match".into(),
        error: RegexError { offset: 1, kind: RegexErrorKind::AmbiguousQuantifier },
    },
);

// `end` back-references name `begin` groups, and there is only one.
let err = Grammar::new("source.demo").pattern(BeginEnd::new("(['\"])", r"\2")).validate().unwrap_err();
assert!(matches!(err, Error::InvalidRegex { ref path, .. } if path == "/patterns/0/end"));

// Two pattern lists that include each other expand forever.
let err = Grammar::new("source.demo")
    .repository("a", Rule::patterns([Rule::include("b")]))
    .repository("b", Rule::patterns([Rule::include("a")]))
    .validate()
    .unwrap_err();
assert_eq!(err.to_string(), "includes form a cycle that consumes no text: #a -> #b -> #a");

Generating from data

Keyword and operator lists usually come from somewhere else; escape_regex makes literal text safe to match:

use textmate_lang::{Grammar, Match, escape_regex};

let keywords = ["let", "fn", "match"];
let operators = ["=>", "->", "::", "+="].map(escape_regex);
let json = Grammar::new("source.demo")
    .pattern(Match::new(format!(r"\b(?:{})\b", keywords.join("|"))).scope("keyword.demo"))
    .pattern(Match::new(operators.join("|")).scope("keyword.operator.demo"))
    .to_json_compact()?;
assert!(json.contains(r#""match":"\\=\\>|\\-\\>|\\:\\:|\\+\\=""#));
# Ok::<(), textmate_lang::Error>(())

Examples

Runnable programs in examples/:

Performance

Validation is one pass over the grammar plus one sort of the repository; includes resolve by binary search over a contiguous name index, and the include graph is searched once, iteratively. The regex checker is a single left-to-right pass with an explicit group stack, reused across a grammar's patterns. Emission writes straight into one String from a work stack, copying unescaped runs of each string in bulk.

Measured with the benchmarks in benches/, Windows x86_64, Rust stable, release profile. Throughput is relative to the size of the pretty JSON (for the regex rows, of the pattern or text):

A grammar the size real languages have validates and emits in well under a millisecond. On the 100,000-entry grammar, include resolution dominates validation: each include is a binary search over 100,000 names that no longer fit in cache. A hash index would be faster there, but its worst case on hostile names is quadratic; the binary search's is not.

cargo bench --bench bench

Criterion writes per-benchmark reports to target/criterion/. Numbers vary by CPU and between runs on this machine by about ±10%; use the trend, not a single absolute.

Design notes

• A subset, on purpose. Oniguruma accepts many spellings that other engines refuse or read differently: x{3}? is an optional x{3}, not a lazy one; plain groups stop being numbered when named groups appear unless the loader asks otherwise (VS Code does, others do not). Refusing these keeps a grammar's meaning the same in every editor. The subset is listed in docs/API.md.
• VS Code's quirks are checked, not assumed away. VS Code substitutes begin captures into end and while patterns by searching the text for a backslash and digits, so \\1 there is misread; it is refused. A top-level | in a selector silently drops the rest; it is refused. + in a selector's scopes is dropped by its tokenizer; it is refused.
• Which cycles are errors. Recursion through a region (parentheses within parentheses) consumes text at every level and is how nested constructs are written. A cycle of pattern lists that include each other consumes nothing: the tokenizer expands the lists while building its scanner and never stops. Only the second is refused.
• Escaping is not conditional on validation. Every string goes through the JSON escaper whatever the validator decided, so a gap in validation could produce a grammar an editor rejects, never JSON with a different structure.
• No recursion where input reaches. Validation and emission run from explicit work stacks; dropping a rule moves nested rules onto a stack; nesting is capped at levels editors handle.

Testing

The suite runs on Windows, Linux, and macOS through the CI matrix, on stable and the 1.85 MSRV:

cargo test                       # unit + integration + property + doctests
cargo test --no-default-features # no_std + alloc
cargo clippy --all-targets --all-features -- -D warnings
cargo bench --bench bench

The property tests in tests/proptests.rs hold the emitter to a strict RFC 8259 reader and a direct construction of the expected JSON, the regex checker to random syntax trees of the subset, the cycle check to a reachability computation, and the scope checker to a direct reading of the naming rules. PROPTEST_CASES=20000 runs a longer soak. Every rust example in this README and in docs/API.md is compiled and run as a doctest.

What the tests cannot show: that every editor's loader accepts every grammar emitted. The subset is chosen from Oniguruma's documented syntax and VS Code's loader source; running emitted grammars through vscode-textmate itself is planned for 0.9 (ROADMAP).

Cross-platform support

• Linux (x86_64, aarch64)
• macOS (x86_64, Apple Silicon)
• Windows (x86_64)

The crate uses no operating-system facilities and no platform-specific code; a grammar emits the same bytes on every platform.

Contributing

See REPS.md and dev/DIRECTIVES.md for the engineering standards every change is held to, and dev/ROADMAP.md for what comes next. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.