Expand description
§treesitter_lang
Generate tree-sitter grammars from Rust:
describe a language’s syntax with Grammar and Rule, and emit the
grammar.js (or grammar.json) that tree-sitter generate turns into a
parser for syntax highlighting, code navigation, and editor tooling.
A language built with the -lang family already has a parser: a
hand-written one that builds a lossless syntax_lang::Node tree. Editors
want a tree-sitter grammar for the same language. This crate keeps the two
in step. The grammar is written in Rust, next to the parser, where it can be
generated from data, shared between dialects, and checked at build time.
And Grammar::sexp renders the trees the hand-written parser builds the
way tree-sitter prints its own, so every sample the parser handles becomes a
tree-sitter corpus test.
§The pieces
Rule— tree-sitter’s rule language:symbol,string,pattern,seq,choice,optional,repeat,field,alias,token,prec, and the rest, one constructor per DSL function.Grammar— the named rules, the first being the start rule, plus extras, externals, supertypes, inlined rules, conflicts, and the word token.Grammar::to_jsandGrammar::to_jsonvalidate it and emit it;Grammar::sexprenders a tree for a corpus test.Error— what validation found: an undefined symbol, a rule other rules refer to that can match nothing, a symbol inside a token, and the other mistakestree-sitter generatewould otherwise stop on.
§Example
use treesitter_lang::{Grammar, Rule};
let e = || Rule::symbol("_expression");
let calc = Grammar::new("calc")
.extra(Rule::pattern(r"\s"))
.extra(Rule::symbol("comment"))
.rule("source_file", Rule::repeat(e()))
.rule("_expression", Rule::choice([
Rule::symbol("number"),
Rule::symbol("binary"),
Rule::seq([Rule::string("("), e(), Rule::string(")")]),
]))
.rule("binary", Rule::choice([
Rule::prec_left(1, Rule::seq([
Rule::field("left", e()),
Rule::field("operator", Rule::choice([Rule::string("+"), Rule::string("-")])),
Rule::field("right", e()),
])),
Rule::prec_left(2, Rule::seq([
Rule::field("left", e()),
Rule::field("operator", Rule::choice([Rule::string("*"), Rule::string("/")])),
Rule::field("right", e()),
])),
]))
.rule("number", Rule::pattern(r"\d+(\.\d+)?"))
.rule("comment", Rule::token(Rule::seq([Rule::string("#"), Rule::pattern(".*")])));
let js = calc.to_js()?;
assert!(js.contains("module.exports = grammar({"));
assert!(js.contains(" field('operator', choice('+', '-')),"));
// std::fs::write("tree-sitter-calc/grammar.js", js)?;§What is checked
Emitting validates the grammar and refuses, with an Error, what
tree-sitter would refuse — or crash or hang on: names that are not
identifiers, a grammar with no rules or a hidden start rule, duplicate
rules, symbols that name nothing, empty strings and patterns, patterns
that are not regular expressions both JavaScript and tree-sitter accept,
symbols inside a token, rules (and repetitions) that can match nothing
where tree-sitter needs a token, cycles of rules that can each be just the
next, and word, external, inline, and supertype declarations tree-sitter
cannot honour. Each check was probed against the tree-sitter CLI, and the whole
was cross-checked by running thousands of random grammars through both.
What validation does not do is build parse tables: conflicts are found by
tree-sitter generate, and declared with Grammar::conflict.
§Features
std(default) — the standard library. Without it the crate isno_stdand needs onlyalloc. Forwards tosyntax-lang/std.
§Re-exports
syntax_lang is re-exported whole: Grammar::sexp takes its
Node, and the re-export lets a caller build trees
with the same version this crate was compiled against.
§Stability
The public surface is frozen and stable as of 1.0.0: it follows Semantic
Versioning, with no breaking changes before 2.0. The full surface, the
validation and output contracts that are part of it, and the SemVer
promise are catalogued in
docs/API.md.
Re-exports§
pub use syntax_lang;
Structs§
- Grammar
- A tree-sitter grammar, built in Rust and emitted as
grammar.jsorgrammar.json. - Rule
- A rule expression: what one rule of a grammar matches.