willbooster-tree-sitter

Rust bindings to this fork of the Tree-sitter parsing library. The crate keeps the library name tree_sitter, so depending on it under the name tree-sitter (as below) needs no code changes.

Basic Usage

First, create a parser:

use tree_sitter::{InputEdit, Language, Parser, Point};

let mut parser = Parser::new();

Then, add a language as a dependency:

[dependencies]
tree-sitter = { package = "willbooster-tree-sitter", version = "2" }
tree-sitter-rust = { package = "willbooster-tree-sitter-rust", version = "3" }

The grammar must be one generated by the CLI of this runtime's major version. The library rejects every other parser: the ones upstream grammar crates ship, and the willbooster-tree-sitter-* releases generated for runtime 1, such as willbooster-tree-sitter-rust 2. Cargo cannot detect the mismatch, because a grammar crate does not depend on the runtime; it shows as an error from set_language.

To use a language, you assign them to the parser.

parser.set_language(&tree_sitter_rust::LANGUAGE.into()).expect("Error loading Rust grammar");

Now you can parse source code:

let source_code = "fn test() {}";
let mut tree = parser.parse(source_code, None).unwrap();
let root_node = tree.root_node();

assert_eq!(root_node.kind(), "source_file");
assert_eq!(root_node.start_position().column, 0);
assert_eq!(root_node.end_position().column, 12);

Editing

Once you have a syntax tree, you can update it when your source code changes. Passing in the previous edited tree makes parse run much more quickly:

let new_source_code = "fn test(a: u32) {}";

tree.edit(&InputEdit {
  start_byte: 8,
  old_end_byte: 8,
  new_end_byte: 14,
  start_position: Point::new(0, 8),
  old_end_position: Point::new(0, 8),
  new_end_position: Point::new(0, 14),
});

let new_tree = parser.parse(new_source_code, Some(&tree));

Text Input

The source code to parse can be provided either as a string, a slice, a vector, or as a function that returns a slice. The text can be encoded as either UTF8 or UTF16:

// Store some source code in an array of lines.
let lines = &[
    "pub fn foo() {",
    "  1",
    "}",
];

// Parse the source code using a custom callback. The callback is called
// with both a byte offset and a row/column offset.
let tree = parser.parse_with(&mut |_byte: usize, position: Point| -> &[u8] {
    let row = position.row as usize;
    let column = position.column as usize;
    if row < lines.len() {
        if column < lines[row].as_bytes().len() {
            &lines[row].as_bytes()[column..]
        } else {
            b"\n"
        }
    } else {
        &[]
    }
}, None).unwrap();

assert_eq!(
  tree.root_node().to_sexp(),
  "(source_file (function_item (visibility_modifier) (identifier) (parameters) (block (number_literal))))"
);

Using Wasm Grammar Files

Requires the feature wasm to be enabled.

First, create a parser with a Wasm store:

use tree_sitter::{wasmtime::Engine, Parser, WasmStore};

let engine = Engine::default();
let store = WasmStore::new(&engine).unwrap();

let mut parser = Parser::new();
parser.set_wasm_store(store).unwrap();

Then, load the language from a Wasm file that the CLI of this runtime's major version built from a parser it generated (tree-sitter generate, then tree-sitter build --wasm); other Wasm files are rejected like other parsers:

const JAVASCRIPT_GRAMMAR: &[u8] = include_bytes!("path/to/tree-sitter-javascript.wasm");

let mut store = WasmStore::new(&engine).unwrap();
let javascript = store
    .load_language("javascript", JAVASCRIPT_GRAMMAR)
    .unwrap();

// The language may be loaded from a different WasmStore than the one set on
// the parser but it must use the same underlying WasmEngine.
parser.set_language(&javascript).unwrap();