Crate weavatrix_parse 

Source tokenizer and structural extractor for repository intelligence.

This crate exists because line-oriented scanning is wrong in ways that matter: a route written inside a comment becomes an endpoint, a // inside a string ends the line early, a declaration spanning three lines disappears, and a class body yields no methods. Every one of those is a tokenizer problem, so this crate starts with a tokenizer.

It deliberately does not build an expression tree. Repository intelligence consumes declarations, imports, exports, calls and their spans - not operator precedence - so the structural pass walks the token stream and stops there. That keeps the crate small enough to own outright and fast enough to run over every file of a monorepo.

No dependencies, no generated grammars, no C, unsafe forbidden.

Re-exports

pub use facts::Contract;

pub use facts::ContractKind;

pub use facts::Declaration;

pub use facts::DeclarationKind;

pub use facts::Facts;

pub use facts::GraphqlOperation;

pub use facts::GraphqlType;

pub use facts::Import;

pub use facts::ImportBinding;

pub use facts::ParseDiagnostic;

pub use facts::Reference;

pub use facts::ReferenceKind;

pub use facts::Span;

pub use syntax::Language;

pub use syntax::Syntax;

pub use token::Mode;

pub use token::Token;

pub use token::TokenKind;

pub use token::Tokenizer;

pub use token::tokenize;

pub use token::tokenize_lite;

Modules

braced

Structural extraction for the brace-scoped languages.

docs

Structural extraction for Markdown, MDX, reStructuredText and AsciiDoc.

facts

Language-neutral facts a structural pass extracts from a token stream.

graphql

Lossless GraphQL SDL and executable-document contract facts.

hcl

Structural extraction for Terraform and HCL.

markup

Structural extraction for HTML and single-file components.

protobuf

Lossless proto3 package, message, service, and RPC contract facts.

python

Structural extraction for Python.

script

Structural extraction for JavaScript and TypeScript.

shell

Structural extraction for shell scripts.

sql

Structural extraction for SQL.

style

Structural extraction for CSS, SCSS, Sass and Less.

syntax

Lexical shape of each supported language.

token

Lossless tokenizer.

Functions

extract

Extracts structural facts from one source file.

extract_path

Extracts structural facts from a file, choosing the language by extension.