Crate textmate_lang 

textmate_lang

Generate TextMate grammars (tmLanguage.json) from Rust: describe a language’s tokens with Grammar and Rules, and emit the JSON that VS Code, Sublime Text, TextMate, Shiki, and other editors and highlighters load to highlight it, no tree-sitter required.

Emitting validates everything a grammar file cannot check for itself. Every regular expression is held to a documented subset of Oniguruma syntax, and refused with a byte offset and a reason when it strays outside. Scope names, scope selectors, captures, and includes are checked; includes that would send a tokenizer into endless recursion are found. The JSON writer is the crate’s own and escapes every string it writes, so no name, pattern, or scope can change the structure of the output.

The pieces

• Grammar — the scope name, top-level patterns, the repository of named rules, and injections. Grammar::to_json validates it and returns the JSON.
• Rule — an entry of a pattern list: a Match, a BeginEnd or BeginWhile region, an include (Rule::include and friends), or a list (Rule::patterns). Capture says what to do with a capture group.
• Error — what validation found, and where, as a JSON Pointer into the output.
• validate_regex and escape_regex — the regular-expression check on its own, and literal text made safe to match.

Example

use textmate_lang::{BeginEnd, Grammar, Match, Rule, escape_regex};

let operators = ["+", "-", "*", "/", "=="].map(escape_regex).join("|");
let json = Grammar::new("source.calc")
    .name("Calc")
    .file_type("calc")
    .pattern(Rule::include("comment"))
    .pattern(Match::new(r"\b\d+(\.\d+)?\b").scope("constant.numeric.calc"))
    .pattern(Match::new(operators).scope("keyword.operator.calc"))
    .pattern(
        Match::new(r"\b(let)\s+([a-z_]\w*)")
            .capture(1, "storage.type.calc")
            .capture(2, "variable.other.calc"),
    )
    .repository(
        "comment",
        BeginEnd::new("#", "$")
            .scope("comment.line.number-sign.calc")
            .begin_capture(0, "punctuation.definition.comment.calc"),
    )
    .to_json()?;

assert!(json.contains(r#""match": "\\+|\\-|\\*|\\/|\\=\\=""#));
// std::fs::write("syntaxes/calc.tmLanguage.json", json)?;

What is checked

• Regular expressions (match, begin, end, while, firstLineMatch) are non-empty and in the subset docs/API.md documents: balanced groups and classes, known escapes, quantifiers with one meaning in every engine, fixed-length look-behind, back-references to groups that exist (in end and while, to groups of begin), and no mixing of named and numbered groups.
• Scope names are dot-separated segments of ASCII letters, digits, _, +, and -; rule names may list several scopes and substitute capture text with $n, which must name an existing group.
• Captures name groups the pattern has, each at most once.
• Includes of repository entries resolve; includes that form a cycle of pattern lists, which tokenizers expand forever, are refused, while recursion through a region (how nested constructs are written) is accepted.
• Selectors (injectionSelector, injection keys) parse.
• Rules nest at most MAX_RULE_DEPTH deep and patterns at most MAX_REGEX_DEPTH; validation and emission are iterative.

What cannot be checked here: rules in other grammars that external includes name, and whether a pattern matches what its author intended.

Features

• std (default) — the standard library. Without it the crate is no_std and needs only alloc. There are no dependencies.

Structs

BeginEnd

A region from a begin match to an end match ("begin"/"end").

BeginWhile

A region that opens at a begin match and continues, line by line, for as long as a while pattern matches at the start of each following line ("begin"/"while"), as Markdown block quotes and lists do.

Capture

What to do with the text a capture group matched: give it a scope, run patterns over it, or both.

Grammar

A TextMate grammar, built in Rust and emitted as tmLanguage.json.

Match

A rule that matches one regular expression ("match").

RegexError

Why a pattern is outside the accepted subset, and where.

Rule

One entry of a pattern list: what the tokenizer tries at each position.

Enums

Error

Why Grammar refused to emit a grammar.

RegexErrorKind

What is wrong with a pattern. See RegexError.

Style

How Grammar::write_json lays out its output.

Constants

MAX_REGEX_DEPTH

How deep groups and character classes may nest in one pattern.

MAX_RULE_DEPTH

How deep rules may nest: regions inside regions, captures with patterns, and pattern lists inside either. Includes do not count.

Functions

escape_regex

Escapes text so that it is a regular expression matching exactly text, in every context a grammar uses: plain and extended ((?x)) mode, and end/while patterns.

validate_regex

Checks that pattern is in the accepted Oniguruma subset and returns its number of capture groups.