weavatrix-parse

Source tokenizer and structural extractor for repository intelligence, and the parsing layer the Weavatrix engine is built on. No dependencies, no generated grammars, no C toolchain, unsafe forbidden.

Why this exists

Line-oriented scanning is wrong in ways that change answers. A route written inside a comment becomes an endpoint. A // inside a string ends the line early. A declaration spanning three lines disappears. A class body yields no methods. Every one of those is a tokenizer problem, so this crate starts with a tokenizer instead of pattern-matching lines.

The alternative was to depend on tree-sitter, which means a C grammar and a build step per language. For a tool whose whole argument is supply-chain clarity, that is the wrong trade — so the tokenizer is owned outright, and the cost of owning it is measured against tree-sitter rather than asserted.

Lossless by default

Every byte of the input belongs to exactly one token, whitespace and comments included, so concatenating the token texts reproduces the source exactly. The test suite asserts that invariant. It is what lets the same stream serve a formatter, a source-to-source translator or a compiler front end — retrofitting losslessness later is expensive, so it is designed in from the start.

Two modes select how much the stream carries:

Spans stay byte-exact in both modes; only trivia presence differs.

Languages

Lexical rules are data rather than a hand-written scanner per language, so one tokenizer stays correct for all of them, and the differences that actually matter are stated explicitly: nested block comments in Rust, raw strings with hash delimiters, triple-quoted Python strings, SQL's doubled-quote escape, significant indentation, whether a slash opens a regular expression or divides, and whether a quote opens a character literal or a lifetime.

Structural extraction — declarations, imports, references and typed transport contracts with byte spans — covers the following languages and formats:

YAML is tokenized but has no structural model yet.

Python f-string expressions are extracted as executable code, including their calls, while ordinary string text and doubled literal braces remain prose.

Swift type heritage is taken from colon lists on types only. The first type on a class inherits; later types, and every colon type on a struct, enum, protocol or extension, implement. Parameter labels such as pairing: Pairing are not heritage. Interpolated strings also yield route fragments: "\(base)/pair/\(mailbox)" produces /pair. Assignments such as request.httpMethod = "PUT" and comps.path = "/ws" are recorded as call facts.

Facts::test_only_declarations carries the exact spans of Rust declarations that exist only in a test compilation. It is derived from syntax, not from a filename: #[test], async test attributes and positive #[cfg(test)] expressions mark the declaration, and an inline #[cfg(test)] mod tests passes that classification to everything nested inside it. #[cfg(not(test))] remains production code. The sparse index keeps the common declaration shape unchanged for every language while allowing consumers to exclude test helpers from dead-code and architecture review without hiding ordinary declarations that happen to share a file with tests.

What this crate does not decide

It extracts what the source says, not what a framework means by it. mongoose.model("User", schema) comes out as a call with receiver mongoose, name model and string argument User; @KafkaListener(topics = "orders") as a call named KafkaListener with argument orders; modelBuilder.Entity<Order>().ToTable("orders") as a use of the type Order beside a call carrying orders. Turning those into "this file declares a collection", "this file subscribes to a topic" or "this entity maps to that table" is framework knowledge — it changes with library versions and there are hundreds of libraries — so it belongs in the consumer, and this crate stays a language layer that does not need updating when a library does.

React is not a separate language here but is the case where the lexer's assumptions are most fragile, so it is pinned by test: inside JSX a / must stay a division rather than opening a regular expression that would swallow the rest of the file, and < must not be read as a comparison.

Several of these earn their place by producing an edge no other extractor can. A stylesheet declares selectors and a document's class and id attributes use them, so the two resolve to each other — and nesting is read from tokens rather than flat rules, so .card { &__title { } } declares .card__title, a name that appears nowhere in the source as written. Terraform ties infrastructure into the same graph: a module names another directory, and every var.x, module.m.out and aws_s3_bucket.b.id is a reference between declared objects. Documents contribute their heading tree and every link that points at a path in the repository rather than at the web.

Vue and Svelte components are read through their script and style blocks, because a component keeps its imports there and nowhere else. Claiming the extension while reading only the template would make the file a graph node with no dependencies — which reads as "this component imports nothing" rather than as "unsupported", and that is worse than not claiming it.

The brace-scoped languages share one walk driven by keyword tables, so adding one of them costs a table rather than a scanner: Solidity was added for two tables and a test. A language with its own scoping model costs a module, as Python and SQL each have.

Enforced modular architecture

The tokenizer is not a growing switch statement. The crate has explicit inward and outward layers:

fact model
    |
lossless lexer
  syntax profiles · token scanner · structural token index
    |
language adapters
  braced · Python · SQL · GraphQL · Protobuf · HCL
  markup · styles · documents · shell
    |
public facade

Each language directory uses one foo/mod.rs module form and is divided by grammar responsibility rather than arbitrary numbered chunks. .weavatrix/architecture.json verifies the dependency direction against the parser's own source graph.

Release gates require zero runtime cycles, files no larger than 300 physical lines, functions no larger than 100 physical lines, no architecture exceptions, an empty baseline, strict Clippy, lossless round trips, exact fact fixtures, and benchmark-tool compilation.

Measured against tree-sitter

Same immutable input, same process and interleaved order for both sides. After one warm-up, the table reports the median of seven measured rounds. Each language is capped at 8 MiB so one vendored tree cannot dominate. extract is tokens plus facts; ts walk is tree-sitter parsing plus one traversal, which is the cheapest way a tree-sitter consumer can reach structural facts. Full method and caveats are in docs/comparison.md.

GraphQL, protobuf, C#, Swift and Terraform had no files in this selected corpus, so there is no throughput result for them. GraphQL and protobuf correctness is instead covered by exact typed fixtures. Protobuf accepts proto2, proto3, and Editions 2023/2024 (including Edition 2024 import option) while preserving every byte and extracting typed package/message/enum/service/RPC facts. The measured code-language range is currently 7.40x to 11.31x, not 30x; a 30x target remains unfulfilled. The machine-readable table is checked in as benchmark-results/competitor-median-2026-07-29.txt (SHA-256 531F03368D07881E4227FDC22A779B724795DF3E117754AAC00B3F32A6187FC0).

Markdown is the one exception to the second rule, and it is not a result to be proud of: prose has no token structure, so the document extractor reads lines directly and never tokenizes at all. It is fast because it does far less than tree-sitter does, not because it does the same thing faster.

Speed is worth nothing if the facts are wrong, so the same corpus is compared on what each side finds. Imports are the fact to compare on, because every grammar marks them with a dedicated node type.

This proves zero misses against tree-sitter's dedicated import nodes on this corpus. It does not by itself prove that every surplus fact is correct: require() and type-position imports are expected examples, while the full surplus remains subject to source review.

The comparison earns its keep by finding defects, not by producing a table. Every one of these was found by running it and read in the source before being fixed:

A character literal holding a quote — ['.', '"', '+'] — left the tokenizer treating that double quote as opening a string, which ran on for hundreds of lines and swallowed every declaration after it. pub(crate) use x; was invisible because the import path stepped over modifiers but not over a parenthesised visibility scope. #include <stdio.h> scanned past the end of its line and consumed the function beneath it. int add(int a, int b) { } matched no declaration rule and fell through to the call path, so every C function definition was recorded as a call to itself. And import service, { helper } from './x' was dropped by a guard that broke at any brace which was not the second token — the one shape a default name before the brace produces.

Rust went 97.2% → 100%, JavaScript and TypeScript 99.9% and 99.7% → 100%.

Reproduce with tools/competitor-bench, a workspace kept outside the published crate so tree-sitter's C grammars never reach it:

cargo run --release --manifest-path tools/competitor-bench/Cargo.toml -- \
  --output target/competitor.txt <corpus-dir>...

What this does not do yet

It does not build a syntax tree. Facts are enough for a dependency graph and are not enough for a compiler, for deciding which parts of a JavaScript codebase can move to WebAssembly, or for translating Go to Rust — all of which need expression structure. That is a third mode on the same tokenizer and is being built; it is not claimed as done.

Nor does it reparse incrementally, or offer a query language for shapes the crate author did not anticipate. tree-sitter does both, and does them well. docs/comparison.md states the gaps without softening them.

License

MIT