Skip to main content

treesitter_lang/
lib.rs

1//! # treesitter_lang
2//!
3//! Generate [tree-sitter](https://tree-sitter.github.io/) grammars from Rust:
4//! describe a language's syntax with [`Grammar`] and [`Rule`], and emit the
5//! `grammar.js` (or `grammar.json`) that `tree-sitter generate` turns into a
6//! parser for syntax highlighting, code navigation, and editor tooling.
7//!
8//! A language built with the `-lang` family already has a parser: a
9//! hand-written one that builds a lossless [`syntax_lang::Node`] tree. Editors
10//! want a tree-sitter grammar for the same language. This crate keeps the two
11//! in step. The grammar is written in Rust, next to the parser, where it can be
12//! generated from data, shared between dialects, and checked at build time.
13//! And [`Grammar::sexp`] renders the trees the hand-written parser builds the
14//! way tree-sitter prints its own, so every sample the parser handles becomes a
15//! tree-sitter corpus test.
16//!
17//! ## The pieces
18//!
19//! - [`Rule`] — tree-sitter's rule language: [`symbol`](Rule::symbol),
20//!   [`string`](Rule::string), [`pattern`](Rule::pattern),
21//!   [`seq`](Rule::seq), [`choice`](Rule::choice),
22//!   [`optional`](Rule::optional), [`repeat`](Rule::repeat),
23//!   [`field`](Rule::field), [`alias`](Rule::alias), [`token`](Rule::token),
24//!   [`prec`](Rule::prec), and the rest, one constructor per DSL function.
25//! - [`Grammar`] — the named rules, the first being the start rule, plus
26//!   extras, externals, supertypes, inlined rules, conflicts, and the word
27//!   token. [`Grammar::to_js`] and [`Grammar::to_json`] validate it and emit
28//!   it; [`Grammar::sexp`] renders a tree for a corpus test.
29//! - [`Error`] — what validation found: an undefined symbol, a rule other
30//!   rules refer to that can match nothing, a symbol inside a token, and the
31//!   other mistakes `tree-sitter generate` would otherwise stop on.
32//!
33//! ## Example
34//!
35//! ```
36//! use treesitter_lang::{Grammar, Rule};
37//!
38//! let e = || Rule::symbol("_expression");
39//! let calc = Grammar::new("calc")
40//!     .extra(Rule::pattern(r"\s"))
41//!     .extra(Rule::symbol("comment"))
42//!     .rule("source_file", Rule::repeat(e()))
43//!     .rule("_expression", Rule::choice([
44//!         Rule::symbol("number"),
45//!         Rule::symbol("binary"),
46//!         Rule::seq([Rule::string("("), e(), Rule::string(")")]),
47//!     ]))
48//!     .rule("binary", Rule::choice([
49//!         Rule::prec_left(1, Rule::seq([
50//!             Rule::field("left", e()),
51//!             Rule::field("operator", Rule::choice([Rule::string("+"), Rule::string("-")])),
52//!             Rule::field("right", e()),
53//!         ])),
54//!         Rule::prec_left(2, Rule::seq([
55//!             Rule::field("left", e()),
56//!             Rule::field("operator", Rule::choice([Rule::string("*"), Rule::string("/")])),
57//!             Rule::field("right", e()),
58//!         ])),
59//!     ]))
60//!     .rule("number", Rule::pattern(r"\d+(\.\d+)?"))
61//!     .rule("comment", Rule::token(Rule::seq([Rule::string("#"), Rule::pattern(".*")])));
62//!
63//! let js = calc.to_js()?;
64//! assert!(js.contains("module.exports = grammar({"));
65//! assert!(js.contains("      field('operator', choice('+', '-')),"));
66//! // std::fs::write("tree-sitter-calc/grammar.js", js)?;
67//! # Ok::<(), treesitter_lang::Error>(())
68//! ```
69//!
70//! ## What is checked
71//!
72//! Emitting validates the grammar and refuses, with an [`Error`], what
73//! tree-sitter would refuse — or crash or hang on: names that are not
74//! identifiers, a grammar with no rules or a hidden start rule, duplicate
75//! rules, symbols that name nothing, empty strings and patterns, patterns
76//! that are not regular expressions both JavaScript and tree-sitter accept,
77//! symbols inside a token, rules (and repetitions) that can match nothing
78//! where tree-sitter needs a token, cycles of rules that can each be just the
79//! next, and word, external, inline, and supertype declarations tree-sitter
80//! cannot honour. Each check was probed against the tree-sitter CLI, and the whole
81//! was cross-checked by running thousands of random grammars through both.
82//! What validation does not do is build parse tables: conflicts are found by
83//! `tree-sitter generate`, and declared with [`Grammar::conflict`].
84//!
85//! ## Features
86//!
87//! - `std` (default) — the standard library. Without it the crate is `no_std`
88//!   and needs only `alloc`. Forwards to `syntax-lang/std`.
89//!
90//! ## Re-exports
91//!
92//! [`syntax_lang`] is re-exported whole: [`Grammar::sexp`] takes its
93//! [`Node`](syntax_lang::Node), and the re-export lets a caller build trees
94//! with the same version this crate was compiled against.
95//!
96//! ## Stability
97//!
98//! The public surface is frozen and stable as of `1.0.0`: it follows Semantic
99//! Versioning, with no breaking changes before `2.0`. The full surface, the
100//! validation and output contracts that are part of it, and the SemVer
101//! promise are catalogued in
102//! [`docs/API.md`](https://github.com/jamesgober/treesitter-lang/blob/main/docs/API.md#stability).
103
104#![cfg_attr(not(feature = "std"), no_std)]
105#![cfg_attr(docsrs, feature(doc_cfg))]
106#![forbid(unsafe_code)]
107#![deny(
108    warnings,
109    missing_docs,
110    unsafe_op_in_unsafe_fn,
111    unused_must_use,
112    unused_results,
113    clippy::unwrap_used,
114    clippy::expect_used,
115    clippy::panic,
116    clippy::todo,
117    clippy::unimplemented,
118    clippy::unreachable,
119    clippy::dbg_macro,
120    clippy::print_stdout,
121    clippy::print_stderr,
122    clippy::undocumented_unsafe_blocks
123)]
124
125extern crate alloc;
126
127mod check;
128mod error;
129mod grammar;
130mod js;
131mod json;
132mod pattern;
133mod rule;
134mod sexp;
135mod text;
136
137pub use error::Error;
138pub use grammar::Grammar;
139pub use rule::Rule;
140
141// The tree type `Grammar::sexp` reads, and the builder that makes it, at the
142// exact version this crate uses.
143pub use syntax_lang;
144
145/// Compiles and runs the `rust` code blocks in `README.md` and `docs/API.md` as
146/// part of `cargo test`, so the published examples cannot drift from the API.
147///
148/// Present only while collecting doctests (`#[cfg(doctest)]`); it is not part of
149/// the public surface and does not appear in the built library or its docs.
150#[cfg(doctest)]
151#[doc = include_str!("../README.md")]
152#[doc = include_str!("../docs/API.md")]
153pub struct MarkdownDocTests;