Skip to main content

Crate treebank

Crate treebank 

Source
Expand description

The treebank vocabulary (notes/DESIGN.md §3), as code.

Two tiers with different physics, dictated by what tree-sitter’s supertype mechanism can express:

  • Table tier — real supertype rules threaded through a grammar’s productions. Occurrence-level semantics, enforced at generate time, natively queryable.
  • Facet tier — roles that cross-cut derivations and therefore cannot be supertypes. Shipped as a roles.json manifest per grammar crate (type-level membership) and expanded into concrete alternations at query-load time by expand.

A term’s tier is a property of the grammar, not of the vocabulary: terms listed in either_tier may be delivered by either mechanism, and each grammar picks. The table tier is stronger and is the default; a grammar demotes a term to a facet only when its language partitions the position (Python orders parameters, so _parameter cannot be one alternation without the grammar accepting def f(a=1, b)). Demotion is sound precisely when every member of the term is a concrete node type that occurs nowhere else, since then type-level and occurrence-level membership select the same nodes; see notes/DESIGN.md §3.4.

The vocabulary itself lives in vocabulary/vocabulary.json, embedded here and re-exported to JavaScript by vocabulary/supertypes.js, so the grammars and this crate can never disagree about what the vocabulary is.

Re-exports§

pub use pack::Pack;

Modules§

check
The vocabulary-conformance checker (treebank roles, notes/DESIGN.md §3.3).
expand
Facet query expansion. Facet roles are not grammar rules, so a query like (_callable name: (_name) @n) cannot run against the parser as written; this rewrites every facet pattern into the concrete alternation the manifest defines:
fetch
Fetch a grammar, so using one is a line of code rather than a build step.
node_types
A minimal reading of tree-sitter’s generated src/node-types.json: which named node types exist, and which of them are supertypes (the entries carrying a subtypes list).
pack
Load a Treebank wasm pack and parse with it.
roles
The roles.json facet manifest each grammar crate ships (notes/DESIGN.md §3.1). Facets are type-level: a node type is _callable wherever it occurs. The manifest also carries the grammar’s uncategorised list — every named node outside the vocabulary, each with a reason — so that nothing is silently outside it (§3.3 rule 2).

Structs§

Term
One vocabulary term: a name (always underscore-prefixed) and its one-line definition.
Vocabulary
The closed vocabulary. A grammar may omit terms its language lacks; it may not invent terms.

Functions§

vocabulary
The vocabulary this build of treebank carries. Parsing the embedded JSON cannot fail for a released crate; the unit tests parse it too, so a malformed edit fails cargo test before it can fail a consumer.