# tree-sitter-bundle
A collection of [tree-sitter](https://tree-sitter.github.io) parsers — and their
highlight/injection/locals queries — compiled into a **single Rust crate**, ready
to drop into a text editor, language server, or any tool that needs syntax trees
for many languages.
Each grammar is behind its own Cargo feature, so you only compile (and pay for)
the languages you enable.
```toml
[dependencies]
# just the languages you need...
tree-sitter-bundle = { version = "0.1", features = ["rust", "python", "json"] }
# ...or everything:
# tree-sitter-bundle = { version = "0.1", features = ["full"] }
```
## How it works
This crate is essentially three things glued together by a build script:
1. **A manifest** (`grammars.toml`) listing each grammar repo, a pinned commit,
the C symbol(s) it exports, and file extensions. Pins mirror the
[neurocyte/tree-sitter](https://github.com/neurocyte/tree-sitter) collection.
2. **A fetch script** (`scripts/fetch-grammars.sh`) that vendors grammar sources
into `grammars/<name>/` at the pinned revisions.
3. **`build.rs`**, which for every *enabled + vendored* grammar compiles its
`parser.c` (and `scanner.c` / `scanner.cc`) with the [`cc`] crate, resolves
its queries, and generates the registry the runtime API reads.
The runtime API is a thin layer over the official [`tree-sitter`] and
[`tree-sitter-highlight`] crates — those provide the parsing engine; this crate
just compiles the grammars and hands you ready-to-use handles.
## Quick start
Grammars are fetched **on demand at build time** — just enable the features you
want and build:
```sh
cargo build --features "rust python json"
```
On the first build, `build.rs` `git`-fetches each enabled grammar at its pinned
revision into a cache under `OUT_DIR` and compiles it. Subsequent builds reuse the
cache. (Requires `git` on `PATH` and network on the first build.)
Prefer to vendor ahead of time (offline, CI, reproducible, or publishing)? Run the
script instead, then build with fetching disabled:
```sh
./scripts/fetch-grammars.sh rust python json # vendors into grammars/
TS_BUNDLE_NO_FETCH=1 cargo build --features "rust python json"
```
### Build-time fetching, in detail
For each enabled grammar, `build.rs` resolves its source in this order:
1. a committed/vendored copy at `grammars/<name>/` (used if present — offline, reproducible);
2. a previously fetched copy in the cache (keyed by revision);
3. a fresh `git` fetch into the cache.
Knobs (all environment variables):
- `TS_BUNDLE_NO_FETCH=1` — never fetch; use only vendored/cached sources. Cargo's
own offline modes (`cargo build --offline` / `--frozen`) are honored too.
- `TS_BUNDLE_GRAMMAR_CACHE=/path` — cache directory. Defaults to `$OUT_DIR/grammars`
(wiped by `cargo clean`); point it at a stable path to keep grammars across cleans.
If a grammar is enabled but can't be resolved (offline with no cache, or a fetch
fails), it's skipped with a `cargo:warning` rather than failing the build.
> **Publishing note:** because build scripts that hit the network break
> `--offline`, docs.rs, and crates.io norms, a published release should vendor +
> commit the sources (run the fetch script, drop the `grammars/*/` ignore) and
> leave fetching off. On-demand fetch is meant for path/git dependencies in your
> own workspace.
### Parsing
```rust
let lang = tree_sitter_bundle::get("rust").unwrap();
let mut parser = tree_sitter::Parser::new();
parser.set_language(&lang.language()).unwrap();
let tree = parser.parse("fn main() {}", None).unwrap();
assert_eq!(tree.root_node().kind(), "source_file");
```
### Detect language by file
```rust
let lang = tree_sitter_bundle::from_path("src/main.rs").unwrap();
assert_eq!(lang.name(), "rust");
```
### Syntax highlighting
```rust
use tree_sitter_highlight::{Highlighter, HighlightEvent};
let names: Vec<String> =
["keyword", "function", "string", "type", "variable"]
.iter().map(|s| s.to_string()).collect();
let lang = tree_sitter_bundle::get("rust").unwrap();
let config = lang.highlight_config(&names).unwrap();
let mut hl = Highlighter::new();
let src = b"fn main() { let x = 42; }";
`injections.scm` / `locals.scm`) and it takes precedence over whatever the
grammar ships. This is how you swap in nvim-treesitter or Helix query sets, which
are usually more consistent than upstream queries.
## Static vs. dynamic loading
This crate takes the **static** approach: parsers are compiled into your binary
(like Zed). Pros: no runtime dependencies, no `dlopen`, single artifact. Cons:
bigger binary, and adding a language is a recompile. If instead you want users to
drop in languages at runtime without rebuilding (like Helix/Neovim, which compile
each grammar to a `.so`/`.dll` and load it dynamically), you'd build the parsers
as cdylibs and load symbols at runtime — a different design than this crate.
## Notes / caveats
- **ABI / runtime version.** The pinned grammars target tree-sitter ABI 15, so
the crate depends on `tree-sitter` 0.25+ (which is why the MSRV is 1.77, coming
from `tree-sitter-language`). If you re-pin grammars to older revisions, lower
the `tree-sitter` dependency to match their ABI.
- **Query correctness varies.** A grammar's bundled queries can be incomplete or
use capture names your renderer doesn't know. Curate per language via the
`queries/` override directory.
- **Symbols for unusual grammars.** A handful of grammars export a symbol that
doesn't follow `tree_sitter_<name>`. The manifest lets you set `symbol` per
language; a few of the rarer single-grammar entries may need one added.
- **Reproducible releases.** Grammar sources are fetched on demand and gitignored.
To publish to crates.io you must vendor and commit the sources (a build script
cannot access the network), then drop the `grammars/*/` ignore.
## License
The crate's own code is MIT. Each vendored grammar and query set keeps its own
upstream license — review them before redistributing.