tree-sitter-bundle

A collection of tree-sitter parsers — and their highlight/injection/locals queries — compiled into a single Rust crate, ready to drop into a text editor, language server, or any tool that needs syntax trees for many languages.

Each grammar is behind its own Cargo feature, so you only compile (and pay for) the languages you enable.

[dependencies]
# just the languages you need...
tree-sitter-bundle = { version = "0.1", features = ["rust", "python", "json"] }
# ...or everything:
# tree-sitter-bundle = { version = "0.1", features = ["full"] }

How it works

This crate is essentially three things glued together by a build script:

1. A manifest (grammars.toml) listing each grammar repo, a pinned commit, the C symbol(s) it exports, and file extensions. Pins mirror the neurocyte/tree-sitter collection.
2. A fetch script (scripts/fetch-grammars.sh) that vendors grammar sources into grammars/<name>/ at the pinned revisions.
3. build.rs, which for every enabled + vendored grammar compiles its parser.c (and scanner.c / scanner.cc) with the [cc] crate, resolves its queries, and generates the registry the runtime API reads.

The runtime API is a thin layer over the official [tree-sitter] and [tree-sitter-highlight] crates — those provide the parsing engine; this crate just compiles the grammars and hands you ready-to-use handles.

Quick start

Grammars are fetched on demand at build time — just enable the features you want and build:

cargo build --features "rust python json"

On the first build, build.rs git-fetches each enabled grammar at its pinned revision into a cache under OUT_DIR and compiles it. Subsequent builds reuse the cache. (Requires git on PATH and network on the first build.)

Prefer to vendor ahead of time (offline, CI, reproducible, or publishing)? Run the script instead, then build with fetching disabled:

./scripts/fetch-grammars.sh rust python json   # vendors into grammars/
TS_BUNDLE_NO_FETCH=1 cargo build --features "rust python json"

Build-time fetching, in detail

For each enabled grammar, build.rs resolves its source in this order:

1. a committed/vendored copy at grammars/<name>/ (used if present — offline, reproducible);
2. a previously fetched copy in the cache (keyed by revision);
3. a fresh git fetch into the cache.

Knobs (all environment variables):

• TS_BUNDLE_NO_FETCH=1 — never fetch; use only vendored/cached sources. Cargo's own offline modes (cargo build --offline / --frozen) are honored too.
• TS_BUNDLE_GRAMMAR_CACHE=/path — cache directory. Defaults to $OUT_DIR/grammars (wiped by cargo clean); point it at a stable path to keep grammars across cleans.

If a grammar is enabled but can't be resolved (offline with no cache, or a fetch fails), it's skipped with a cargo:warning rather than failing the build.

Publishing note: because build scripts that hit the network break --offline, docs.rs, and crates.io norms, a published release should vendor + commit the sources (run the fetch script, drop the grammars/*/ ignore) and leave fetching off. On-demand fetch is meant for path/git dependencies in your own workspace.

Parsing

let lang = tree_sitter_bundle::get("rust").unwrap();
let mut parser = tree_sitter::Parser::new();
parser.set_language(&lang.language()).unwrap();
let tree = parser.parse("fn main() {}", None).unwrap();
assert_eq!(tree.root_node().kind(), "source_file");

Detect language by file

let lang = tree_sitter_bundle::from_path("src/main.rs").unwrap();
assert_eq!(lang.name(), "rust");

Syntax highlighting

use tree_sitter_highlight::{Highlighter, HighlightEvent};

let names: Vec<String> =
    ["keyword", "function", "string", "type", "variable"]
        .iter().map(|s| s.to_string()).collect();

let lang = tree_sitter_bundle::get("rust").unwrap();
let config = lang.highlight_config(&names).unwrap();

let mut hl = Highlighter::new();
let src = b"fn main() { let x = 42; }";
for event in hl.highlight(&config, src, None, |_| None).unwrap() {
    match event.unwrap() {
        HighlightEvent::Source { start, end } => { /* text src[start..end] */ }
        HighlightEvent::HighlightStart(h)     => { /* open span names[h.0]  */ }
        HighlightEvent::HighlightEnd           => { /* close span            */ }
    }
}

API

Each LanguageEntry exposes name(), extensions(), language(), highlights_query(), injections_query(), locals_query(), and (with the highlight feature) highlight_config().

Adding or overriding grammars

Add a grammar: append a [[grammar]] block to grammars.toml and a matching feature line in Cargo.toml. For repos that ship several grammars, list each under [[grammar.language]] with its src subdir and exported symbol:

[[grammar]]
name = "typescript"
repo = "https://github.com/tree-sitter/tree-sitter-typescript"
rev  = "<commit>"
[[grammar.language]]
id = "typescript"
src = "typescript/src"
symbol = "tree_sitter_typescript"
extensions = ["ts"]
[[grammar.language]]
id = "tsx"
src = "tsx/src"
symbol = "tree_sitter_tsx"
extensions = ["tsx"]

Override queries: drop a file at queries/<id>/highlights.scm (or injections.scm / locals.scm) and it takes precedence over whatever the grammar ships. This is how you swap in nvim-treesitter or Helix query sets, which are usually more consistent than upstream queries.

Static vs. dynamic loading

This crate takes the static approach: parsers are compiled into your binary (like Zed). Pros: no runtime dependencies, no dlopen, single artifact. Cons: bigger binary, and adding a language is a recompile. If instead you want users to drop in languages at runtime without rebuilding (like Helix/Neovim, which compile each grammar to a .so/.dll and load it dynamically), you'd build the parsers as cdylibs and load symbols at runtime — a different design than this crate.

Notes / caveats

• ABI / runtime version. The pinned grammars target tree-sitter ABI 15, so the crate depends on tree-sitter 0.25+ (which is why the MSRV is 1.77, coming from tree-sitter-language). If you re-pin grammars to older revisions, lower the tree-sitter dependency to match their ABI.
• Query correctness varies. A grammar's bundled queries can be incomplete or use capture names your renderer doesn't know. Curate per language via the queries/ override directory.
• Symbols for unusual grammars. A handful of grammars export a symbol that doesn't follow tree_sitter_<name>. The manifest lets you set symbol per language; a few of the rarer single-grammar entries may need one added.
• Reproducible releases. Grammar sources are fetched on demand and gitignored. To publish to crates.io you must vendor and commit the sources (a build script cannot access the network), then drop the grammars/*/ ignore.

License

The crate's own code is MIT. Each vendored grammar and query set keeps its own upstream license — review them before redistributing.