weavatrix-scan 0.2.0

Deterministic, safe repository scanner for code intelligence
Documentation

Weavatrix Scan

CI crates.io docs.rs MIT MSRV

weavatrix-scan is a deterministic, read-only repository scanner for static analysis, code intelligence, indexing, and AI tooling.

It does more than walk a directory. A scan produces a stable manifest with normalized paths, file sizes, optional content hashes, an aggregate revision, and explicit evidence explaining why files were skipped. Linux and macOS builds have zero mandatory runtime dependencies; Windows uses only winapi-util for native volume and file identities.

Why another repository walker?

walkdir and jwalk are excellent traversal libraries. ignore adds mature Git-style filtering. Weavatrix Scan now exposes four deliberately separate layers:

  • Walker: iterative, streaming, lossless low-level traversal;
  • Scanner: ignore-aware deterministic manifest, hashes, revision, and typed evidence;
  • RepositoryMatcher: cached path selection for incremental consumers;
  • ParallelWalker: bounded collected or streaming traversal for wide trees.
Capability weavatrix-scan ignore walkdir jwalk
Iterative traversal Yes Yes Yes Yes
Lossless native paths Yes Yes Yes Yes
Continue after local errors Configurable Yes Yes Yes
max_depth / bounded handles Yes Yes Yes Depth limit
Same-filesystem boundary Yes No Yes No
.gitignore hierarchy Yes Yes No No
Custom ignore files Yes Yes No No
Repository / Git-compatible ignore modes Yes Yes No No
Override globs / source switches Yes Yes No Directory callback
Reusable cached matcher Yes Yes No No
Stable normalized paths Yes No No Sorted traversal
File sizes and content hashes Yes No No No
Aggregate deterministic revision Yes No No No
Typed manifest delta / rename evidence Yes No No No
Binary and oversized-file policy Yes No No No
Typed skip reasons and warnings Yes No No No
Symlinks skipped by default / loop detection Yes Yes Yes Configurable
Parallel collected / streaming traversal Yes / Yes Yes / Yes No Yes / Yes
Cancellation and whole-scan budgets Yes Quit only No No
Minimum depth / hidden policy Yes / Yes Yes / Yes Yes / No Yes / Yes
Default runtime dependencies 0 Unix / 1 Windows Multiple 2 platform helpers Rayon stack

Use Walker when you only need paths. Use Scanner when downstream results must be reproducible and explainable.

Install

[dependencies]
weavatrix-scan = "0.2"

Enable serialization only when needed:

[dependencies]
weavatrix-scan = { version = "0.2", features = ["serde"] }

Quick start

use weavatrix_scan::{ScanOptions, Scanner};

let options = ScanOptions::default()
    .with_extensions(["rs", "go", "ts", "py"])
    .with_parallelism(0);

let report = Scanner::new(".").options(options).scan()?;

println!("revision: {}", report.revision);
for file in &report.files {
    println!(
        "{}: {} bytes, hash={}",
        file.relative,
        file.bytes,
        file.content_hash.as_deref().unwrap_or("disabled")
    );
}
for skipped in &report.skipped {
    println!("skipped {}: {:?}", skipped.relative, skipped.kind);
}
# Ok::<(), weavatrix_scan::Error>(())

For the fastest path-only discovery, disable content reads:

use weavatrix_scan::{ScanOptions, Scanner};

let report = Scanner::new(".")
    .options(
        ScanOptions::default()
            .with_extensions(["rs", "go", "ts"])
            .metadata_only()
            .selected_files_only(),
    )
    .scan()?;

assert!(report.files.iter().all(|file| file.content_hash.is_none()));
# Ok::<(), weavatrix_scan::Error>(())

Low-level walkers

Walker is a streaming iterator. It keeps paths as native PathBuf/OsStr values, uses iterative DFS, bounds open directory handles, and yields local errors according to policy:

use weavatrix_scan::{ErrorPolicy, WalkOptions, Walker};

let options = WalkOptions::default()
    .with_max_depth(Some(64))
    .with_max_open(8)
    .with_same_file_system(true)
    .with_error_policy(ErrorPolicy::Continue);

let mut walker = Walker::with_options(".", options)?;
while let Some(item) = walker.next() {
    match item {
        Ok(entry) => println!("{}", entry.path().display()),
        Err(error) => eprintln!("partial walk: {error}"),
    }
}
# Ok::<(), weavatrix_scan::WalkError>(())

The root is depth zero. After receiving a directory, callers can invoke skip_current_dir() before requesting the next item. Symbolic links are not followed by default; enabling .with_follow_links(true) keeps traversal inside the root and reports loops as typed skip reasons.

ParallelWalker is a separate collected mode for wide directory frontiers:

use weavatrix_scan::ParallelWalker;

let report = ParallelWalker::new(".")
    .with_parallelism(0)
    .walk()?;
println!("entries={}, local_errors={}", report.entries.len(), report.errors.len());
# Ok::<(), weavatrix_scan::WalkError>(())

For pipelines that should parse entries immediately instead of collecting them, visit invokes a thread-safe callback directly on traversal workers:

use weavatrix_scan::{ParallelWalker, WalkControl, WalkEvent};

let summary = ParallelWalker::new(".").visit(|event| match event {
    WalkEvent::Entry(entry) if entry.file_name() == "target" => WalkControl::Skip,
    WalkEvent::Entry(entry) => {
        println!("{}", entry.path().display());
        WalkControl::Continue
    }
    WalkEvent::Error(error) => {
        eprintln!("{error}");
        WalkControl::Continue
    }
})?;
# Ok::<(), weavatrix_scan::WalkError>(())

Scan modes

The same scanner supports three useful cost levels:

Mode Configuration Reads content Skip evidence Hashes content
Rich manifest ScanOptions::default() Yes Complete Yes
Safe discovery hash_file_contents = false First 8 KiB Complete No
Metadata only .metadata_only() No Complete No
Selected manifest .metadata_only().selected_files_only() No Omitted No

Content inspection uses available CPU parallelism by default. Set .with_parallelism(1) for a serial run or pass a fixed worker count when a host application owns the wider scheduling policy.

Output contract

ScanReport contains:

  • root: canonical absolute repository root;
  • files: stable, lexicographically sorted ScannedFile values;
  • skipped: stable, sorted evidence for excluded entries;
  • warnings: non-fatal ignore-file and local I/O diagnostics;
  • ignore_sources: typed location and hash of every loaded selection input;
  • revision: FNV-1a digest over ignore inputs, selected paths, optional content hashes, portability, and partial-termination state;
  • complete: false when local errors made the evidence partial.
  • termination: typed reason for a bounded or cancelled partial scan;
  • portable: false when host-level Git configuration affected selection.

Each ScannedFile contains an absolute path, slash-normalized repository path, byte size, and optional content hash. Default hashes are deterministic FNV-1a digests intended for change detection, not cryptographic verification. Native paths remain lossless in the walker and absolute PathBuf; invalid Unicode units in normalized manifest names are escaped (%XX on Unix, %uXXXX on Windows) instead of being replaced with the lossy Unicode replacement marker. With the serde feature, invalid native path units use a tagged byte/wide-unit representation and round-trip without loss; ordinary Unicode paths remain plain JSON strings.

Incremental consumers

Two completed reports produce a stable changed-file set without filesystem access:

use weavatrix_scan::{DeltaQuality, Scanner};

let previous = Scanner::new(".").scan()?;
// Apply repository changes, then scan again.
let current = Scanner::new(".").scan()?;
let delta = current.delta_from(&previous);

assert!(matches!(
    delta.quality,
    DeltaQuality::ContentHash | DeltaQuality::Metadata | DeltaQuality::Partial
));
println!(
    "added={} modified={} removed={} renamed={}",
    delta.added.len(),
    delta.modified.len(),
    delta.removed.len(),
    delta.renamed.len()
);
# Ok::<(), weavatrix_scan::Error>(())

Rename evidence is emitted only when the same content hash is unique in both manifests; duplicate-content moves remain explicit add/remove pairs instead of being guessed. Metadata-only scans classify same-size files as unchanged with DeltaQuality::Metadata, so callers can decide whether to request content hashes. Partial scans always produce DeltaQuality::Partial.

Long-lived file watchers can keep a RepositoryMatcher and call refresh() after an ignore input changes. Refresh builds a replacement matcher first, so a failure leaves the existing matcher usable.

SkipKind distinguishes:

  • Binary
  • Extension
  • FileSystemBoundary
  • Hidden
  • Ignored
  • IoError
  • MaxDepth
  • Override
  • Oversized
  • PathEscape
  • ScanLimit
  • StandardDirectory
  • Symlink
  • SymlinkLoop

This distinction matters to analyzers: "not selected by policy" is different from "unreadable" or "outside the repository."

Configuration

ScanOptions exposes:

Option Default Purpose
max_file_bytes 1,500,000 Reject oversized source candidates
extensions Empty Empty accepts every extension
ignore_files .gitignore, .ignore, .weavatrixignore Hierarchical local ignore files
ignore_policy Repository-only Optional parents, .git/info/exclude, global Git and explicit files
override_rules Empty Request-level include/exclude globs above ignore sources
ignore_case_insensitive false Optional ASCII case-insensitive ignore matching
skip_hidden false Skip dot-prefixed and Windows-hidden entries unless included
standard_skips Enabled Skip generated/vendor directories
hash_file_contents true Attach per-file hashes and content-sensitive revision
detect_binary_files true Reject files containing a NUL byte
evidence Complete Keep all typed exclusions, or only selected files
parallelism 0 Content workers; zero uses available parallelism
limits.max_entries None Bound examined filesystem entries
limits.max_total_bytes None Deterministically bound selected content bytes
limits.timeout None Stop traversal/content inspection after a duration
cancellation None Cooperative cross-thread cancellation token
walk.max_depth None Limit entry depth; root is zero
walk.min_depth 0 Suppress shallower results while still traversing them
walk.max_open 64 Bound live directory handles/workers
walk.same_file_system false Stop at filesystem boundaries when enabled
walk.follow_links false Follow only in-root links and detect cycles
walk.error_policy Continue Continue with partial typed evidence or abort
walk.collect_metadata true in ScanOptions Reuse directory-entry metadata without reopening selected paths

The standard directory policy skips:

.git .hg .svn .venv __pycache__ build coverage dist
node_modules target vendor

Disable it when another layer owns generated-directory policy:

use weavatrix_scan::{ScanOptions, StandardSkips};

let mut options = ScanOptions::default();
options.standard_skips = StandardSkips::Disabled;

Ignore semantics

Ignore files are loaded hierarchically with source precedence .weavatrixignore/custom > .ignore > .gitignore > .git/info/exclude > global Git. Deeper files win within the same source class. Supported Git-style constructs include:

  • comments and escaped leading # / !;
  • negation with !;
  • root-anchored patterns;
  • directory-only patterns;
  • *, **, and ?;
  • character classes, negated classes, and ranges;
  • brace alternatives such as {foo,bar};
  • escaped literals and escaped trailing spaces.

The default scanner intentionally does not read global Git configuration, parent rules outside the scan root, or .git/info/exclude; repository-local selection therefore stays portable. IgnorePolicy::git_compatible() enables all three explicitly inside Git repositories, records their content hashes, and marks host-dependent reports non-portable. Local .gitignore, .ignore, and custom sources can be toggled independently. Request-level override globs use ignore::Override semantics: ordinary patterns include and leading ! patterns exclude. Explicit includes can opt paths back into standard-directory and extension filtering, but never bypass size or binary safety checks. RepositoryMatcher::matched exposes the winning typed decision without requiring a full walk. Differential tests compare exact selected path sets against the ignore crate for anchored, nested, negated, wildcard, and character-class fixtures plus deterministic randomized rule sets. Stress cases cover deep trees, permission errors, non-UTF8 names, and followed symlink loops. The differential suite and competitor crates are dev-only.

Safety model

  • never executes repository code;
  • never starts subprocesses or accesses the network;
  • canonicalizes and validates the root before traversal;
  • does not follow symlink entries by default;
  • rejects followed links outside the canonical root and detects cycles;
  • can enforce a same-filesystem boundary;
  • continues after independent local errors by default and marks the report partial;
  • caps selected file size before content reads;
  • rejects repository-local ignore-file symlinks and path traversal;
  • supports entry, total-byte, timeout, and cooperative cancellation bounds;
  • forbids unsafe Rust.

The scanner is read-only. Concurrent filesystem changes are surfaced as local warnings/skips under Continue or as the first error under Abort.

Benchmarks

Run all included benchmarks:

cargo bench --locked

Run the competitor comparison:

cargo bench --locked --bench compare_competitors

The Competitor benchmarks workflow runs the same output-equivalent comparison on Ubuntu, Windows, and macOS for scanner or benchmark changes.

Run exact selected-path parity on a real repository:

$env:WEAVATRIX_BENCH_ROOT = "C:\path\to\repository"
cargo bench --locked --bench real_repository

The synthetic comparison uses 6,000 source files across Rust, Go, and TypeScript in 80 sibling directories. It runs two warmups and 11 interleaved measured samples, then reports the median. Raw walkers must produce the same fully sorted native relative-path set; the ignore-aware comparison additionally checks the same normalized path-and-size manifest.

Sample result on Windows 11, Rust 1.97.1, warm filesystem cache, measured 2026-07-24 against ignore 0.4.31, walkdir 2.5.0, and jwalk 0.8.1:

Mode Library Files Median
Raw paths weavatrix Walker 6,004 7.7 ms
Raw paths weavatrix ParallelWalker 6,004 5.0 ms
Raw paths ignore 6,004 10.1 ms
Raw paths walkdir 6,004 9.5 ms
Raw paths jwalk 6,004 7.8 ms
Ignore-aware manifest weavatrix Scanner 6,001 20.5 ms
Ignore-aware manifest ignore 6,001 24.1 ms
Rich manifest weavatrix Scanner 6,000 92.8 ms

This is the median of five independent output-equivalent Windows benchmark processes; each process itself reports the median of 11 interleaved samples after two warmups. Walker and ParallelWalker beat walkdir on this corpus. ParallelWalker is about 36% faster than jwalk, while the selected-manifest Scanner is about 15% faster than ignore. The comparable scanner row omits skip evidence on both sides. The rich row additionally records typed evidence, reads content, detects binaries, hashes sources, and computes a deterministic revision.

One cross-platform GitHub-hosted-runner sample on the same commit:

Platform ParallelWalker jwalk walkdir Scanner ignore
Ubuntu 7.2 ms 7.8 ms 9.7 ms 21.1 ms 22.8 ms
Windows 6.5 ms 9.5 ms 11.4 ms 23.5 ms 27.5 ms
macOS 5.6 ms 7.1 ms 8.0 ms 15.2 ms 20.5 ms

Absolute timings are not comparable between runner operating systems because their hardware differs. Within every row, Weavatrix produced identical output and led both the parallel-walker and ignore-aware comparisons.

Source review explains the remaining differences:

  • walkdir streams unsorted directory entries and bounds open descriptors;
  • jwalk schedules read_dir work through Rayon and restores ordered output;
  • ignore compiles patterns into GlobSet matchers and shares inherited matchers;
  • Weavatrix Walker streams iterative DFS, bounds live handles and buffers the oldest remaining frame only when max_open is reached;
  • Weavatrix ParallelWalker keeps round-robin directory balancing but restores discovery-task order independently of worker completion;
  • Weavatrix Scanner reuses inherited rules, indexes exact literals, specializes prefix/suffix globs, prefilters complex patterns, and sorts only the final report.

Exact-path real-repository sample:

Repository Files weavatrix-scan ignore
radiochron 86 18.7 ms 22.9 ms
grpc-server 30 5.4 ms 5.6 ms
bgp-speaker 30 4.7 ms 5.6 ms
controller-rest-api 1,085 38.3 ms 38.4 ms
frontend 1,689 39.8 ms 46.3 ms
analytics 361 40.6 ms 40.4 ms
automation 1,670 22.6 ms 22.2 ms

Every real row first asserts the exact same sorted (normalized path, bytes) manifest. Weavatrix is faster on five repositories; the remaining two are within 2%, below the observed run-to-run variance. Timing varies by filesystem, cache, antivirus, and CPU, so treat the table as a reproducible sample rather than a universal constant.

Correctness checks

The test suite covers:

  • deterministic results and revisions;
  • ignore-rule precedence and nested ignore files;
  • repository-only, Git-exclude, parent, explicit and reusable-matcher policies;
  • representative and randomized parity with ignore;
  • raw entry parity with walkdir and jwalk;
  • iterative deep trees, bounded handles, local error continuation, non-UTF8 paths, and symlink loops;
  • binary, oversized, extension, generated-directory, and symlink policies;
  • serial/parallel content-inspection equivalence;
  • streaming parallel pruning and cancellation;
  • manifest delta evidence and live matcher refresh;
  • optional Serde support.

The real-repository benchmark compares the complete normalized selected-path set against ignore. Its comparison policy disables Weavatrix's file-size cap so an oversized file cannot masquerade as an ignore-rule mismatch.

Development

cargo fmt --all -- --check
cargo test --locked --all-features
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo doc --locked --no-deps --all-features
cargo bench --locked
cargo publish --locked --dry-run

The MSRV is Rust 1.88. CI checks Rust 1.88 on Linux, Windows, and macOS, with stable test coverage on all three platforms.

Relationship to Weavatrix

weavatrix-scan owns repository discovery. It does not parse languages or build graphs. weavatrix-graph owns typed graph primitives. Higher-level Weavatrix crates can compose both without coupling either library to MCP, a CLI, or language-specific parsers.

License

MIT © 2026 Sergii Ziborov.