treesitter_lang/lib.rs
1//! # treesitter_lang
2//!
3//! Generate [tree-sitter](https://tree-sitter.github.io/) grammars from Rust:
4//! describe a language's syntax with [`Grammar`] and [`Rule`], and emit the
5//! `grammar.js` (or `grammar.json`) that `tree-sitter generate` turns into a
6//! parser for syntax highlighting, code navigation, and editor tooling.
7//!
8//! A language built with the `-lang` family already has a parser: a
9//! hand-written one that builds a lossless [`syntax_lang::Node`] tree. Editors
10//! want a tree-sitter grammar for the same language. This crate keeps the two
11//! in step. The grammar is written in Rust, next to the parser, where it can be
12//! generated from data, shared between dialects, and checked at build time.
13//! And [`Grammar::sexp`] renders the trees the hand-written parser builds the
14//! way tree-sitter prints its own, so every sample the parser handles becomes a
15//! tree-sitter corpus test.
16//!
17//! ## The pieces
18//!
19//! - [`Rule`] — tree-sitter's rule language: [`symbol`](Rule::symbol),
20//! [`string`](Rule::string), [`pattern`](Rule::pattern),
21//! [`seq`](Rule::seq), [`choice`](Rule::choice),
22//! [`optional`](Rule::optional), [`repeat`](Rule::repeat),
23//! [`field`](Rule::field), [`alias`](Rule::alias), [`token`](Rule::token),
24//! [`prec`](Rule::prec), and the rest, one constructor per DSL function.
25//! - [`Grammar`] — the named rules, the first being the start rule, plus
26//! extras, externals, supertypes, inlined rules, conflicts, and the word
27//! token. [`Grammar::to_js`] and [`Grammar::to_json`] validate it and emit
28//! it; [`Grammar::sexp`] renders a tree for a corpus test.
29//! - [`Error`] — what validation found: an undefined symbol, a rule other
30//! rules refer to that can match nothing, a symbol inside a token, and the
31//! other mistakes `tree-sitter generate` would otherwise stop on.
32//!
33//! ## Example
34//!
35//! ```
36//! use treesitter_lang::{Grammar, Rule};
37//!
38//! let e = || Rule::symbol("_expression");
39//! let calc = Grammar::new("calc")
40//! .extra(Rule::pattern(r"\s"))
41//! .extra(Rule::symbol("comment"))
42//! .rule("source_file", Rule::repeat(e()))
43//! .rule("_expression", Rule::choice([
44//! Rule::symbol("number"),
45//! Rule::symbol("binary"),
46//! Rule::seq([Rule::string("("), e(), Rule::string(")")]),
47//! ]))
48//! .rule("binary", Rule::choice([
49//! Rule::prec_left(1, Rule::seq([
50//! Rule::field("left", e()),
51//! Rule::field("operator", Rule::choice([Rule::string("+"), Rule::string("-")])),
52//! Rule::field("right", e()),
53//! ])),
54//! Rule::prec_left(2, Rule::seq([
55//! Rule::field("left", e()),
56//! Rule::field("operator", Rule::choice([Rule::string("*"), Rule::string("/")])),
57//! Rule::field("right", e()),
58//! ])),
59//! ]))
60//! .rule("number", Rule::pattern(r"\d+(\.\d+)?"))
61//! .rule("comment", Rule::token(Rule::seq([Rule::string("#"), Rule::pattern(".*")])));
62//!
63//! let js = calc.to_js()?;
64//! assert!(js.contains("module.exports = grammar({"));
65//! assert!(js.contains(" field('operator', choice('+', '-')),"));
66//! // std::fs::write("tree-sitter-calc/grammar.js", js)?;
67//! # Ok::<(), treesitter_lang::Error>(())
68//! ```
69//!
70//! ## What is checked
71//!
72//! Emitting validates the grammar and refuses, with an [`Error`], what
73//! tree-sitter would refuse — or crash or hang on: names that are not
74//! identifiers, a grammar with no rules or a hidden start rule, duplicate
75//! rules, symbols that name nothing, empty strings and patterns, patterns
76//! that are not regular expressions both JavaScript and tree-sitter accept,
77//! symbols inside a token, rules (and repetitions) that can match nothing
78//! where tree-sitter needs a token, cycles of rules that can each be just the
79//! next, and word, external, inline, and supertype declarations tree-sitter
80//! cannot honour. Each check was probed against the tree-sitter CLI, and the whole
81//! was cross-checked by running thousands of random grammars through both.
82//! What validation does not do is build parse tables: conflicts are found by
83//! `tree-sitter generate`, and declared with [`Grammar::conflict`].
84//!
85//! ## Features
86//!
87//! - `std` (default) — the standard library. Without it the crate is `no_std`
88//! and needs only `alloc`. Forwards to `syntax-lang/std`.
89//!
90//! ## Re-exports
91//!
92//! [`syntax_lang`] is re-exported whole: [`Grammar::sexp`] takes its
93//! [`Node`](syntax_lang::Node), and the re-export lets a caller build trees
94//! with the same version this crate was compiled against.
95//!
96//! ## Stability
97//!
98//! The public surface is frozen and stable as of `1.0.0`: it follows Semantic
99//! Versioning, with no breaking changes before `2.0`. The full surface, the
100//! validation and output contracts that are part of it, and the SemVer
101//! promise are catalogued in
102//! [`docs/API.md`](https://github.com/jamesgober/treesitter-lang/blob/main/docs/API.md#stability).
103
104#![cfg_attr(not(feature = "std"), no_std)]
105#![cfg_attr(docsrs, feature(doc_cfg))]
106#![forbid(unsafe_code)]
107#![deny(
108 warnings,
109 missing_docs,
110 unsafe_op_in_unsafe_fn,
111 unused_must_use,
112 unused_results,
113 clippy::unwrap_used,
114 clippy::expect_used,
115 clippy::panic,
116 clippy::todo,
117 clippy::unimplemented,
118 clippy::unreachable,
119 clippy::dbg_macro,
120 clippy::print_stdout,
121 clippy::print_stderr,
122 clippy::undocumented_unsafe_blocks
123)]
124
125extern crate alloc;
126
127mod check;
128mod error;
129mod grammar;
130mod js;
131mod json;
132mod pattern;
133mod rule;
134mod sexp;
135mod text;
136
137pub use error::Error;
138pub use grammar::Grammar;
139pub use rule::Rule;
140
141// The tree type `Grammar::sexp` reads, and the builder that makes it, at the
142// exact version this crate uses.
143pub use syntax_lang;
144
145/// Compiles and runs the `rust` code blocks in `README.md` and `docs/API.md` as
146/// part of `cargo test`, so the published examples cannot drift from the API.
147///
148/// Present only while collecting doctests (`#[cfg(doctest)]`); it is not part of
149/// the public surface and does not appear in the built library or its docs.
150#[cfg(doctest)]
151#[doc = include_str!("../README.md")]
152#[doc = include_str!("../docs/API.md")]
153pub struct MarkdownDocTests;