weavatrix-scan 0.3.0

Deterministic, safe repository scanner for code intelligence
Documentation

Weavatrix Scan

CI crates.io docs.rs MIT MSRV

weavatrix-scan is a deterministic, read-only repository scanner for static analysis, code intelligence, indexing, and AI tooling.

It does more than walk a directory. A scan produces a stable manifest with normalized paths, file sizes, optional content hashes, an aggregate revision, and explicit evidence explaining why files were skipped. Linux and macOS builds have zero mandatory runtime dependencies; Windows uses only winapi-util for native volume and file identities.

Why another repository walker?

walkdir and jwalk are excellent traversal libraries. ignore adds mature Git-style filtering. Weavatrix Scan exposes deliberately separate layers:

  • Walker: iterative, streaming, lossless low-level traversal;
  • WalkBuilder: multi-root traversal, native sorting, directory filters, and contents-first ordering;
  • Scanner: ignore-aware deterministic manifest, hashes, revision, typed evidence, and bounded one-pass selected-content callbacks;
  • CompactScanReport: the same deterministic selection and revision with one retained root path and optional boxed rich evidence for million-file manifests;
  • MultiScanner: ordered concurrent manifests and a globally bounded multi-root content pipeline;
  • ParallelMultiWalker: collected or direct-streaming traversal across concurrent raw roots with ordered per-root reports;
  • RepositoryMatcher: cached path selection for incremental consumers;
  • ParallelWalker: bounded adaptive traversal for broad or skewed trees.
  • ParallelRuntime: process-global, dedicated, or application-owned execution shared by walker and scanner APIs.

Competitive position

Against the versions tested in this repository (ignore 0.4.31, walkdir 2.5.0, and jwalk 0.8.1), Weavatrix Scan is the strongest overall fit when the output must be a deterministic, explainable code-scanner manifest rather than only a stream of directory entries. It is not the universal winner for every walker workload:

Workload Strongest fit in this comparison Why
Deterministic code-scanner manifest Weavatrix Scan Only entry with normalized paths, hashes, aggregate revision, typed skips, portable evidence, incremental cache, and changed-path updates
Raw parallel streaming Weavatrix ParallelWalker 264.7 ms and 7.6 MiB peak on the measured 1,000,000-file Windows fixture
Minimal serial traversal walkdir / Weavatrix Walker walkdir remains the small established primitive; Walker measured 546.3 ms versus 584.5 ms with comparable 5 MiB-class memory
Memory-efficient deterministic manifest Weavatrix CompactScanReport Exact output parity at 1,019.4 ms / 63.2 MiB; ignore used 56.4 MiB but took 2,106.7 ms and does not produce revision/evidence
Git-ignore matcher ecosystem ignore Reference implementation and broader production history; Weavatrix continuously checks exact manifest parity against it
Host-owned parallel scheduling Weavatrix / jwalk Weavatrix accepts any fallible executor plus busy-timeout policy or creates an owned pool; jwalk has direct Rayon pool modes
Capability weavatrix-scan ignore walkdir jwalk
Iterative traversal Yes Yes Yes Yes
Single-file root Yes Yes Yes Yes
Lossless native paths Yes Yes Yes Yes
Continue after local errors Configurable Yes Yes Yes
Depth / open-handle control Depth + configurable max_open Depth + internally bounded Depth + configurable max_open Depth + directory scheduler
Same-filesystem boundary Yes Yes Yes No
.gitignore hierarchy Yes Yes No No
Custom ignore files Yes Yes No No
Repository / Git-compatible ignore modes Yes Yes No No
Override globs / source switches Yes Yes No No
Reusable cached matcher Yes Yes No No
Multi-root / custom sort Serial + parallel / full DirEntry Yes / name or path, serial only No / full DirEntry No / mutable directory batch
Directory callback / contents-first Parallel typed batch / Yes Filter only / No Filter / Yes Parallel typed batch / No
Built-in types / composition / negation 265 / Yes / Yes 224 / Yes / Yes No No
Stable normalized paths Yes No No No
Path-safe portable report Yes No No No
Snapshot-verified content provider Yes No No No
File sizes and SHA-256 hashes Yes No No No
Versioned compact incremental cache Yes No No No
Watcher events to changed-path manifest update Yes No No No
Optional direct notify adapter Yes No No No
Concurrent-mutation evidence Yes No No No
Aggregate deterministic revision Yes No No No
Typed manifest delta / rename evidence Yes No No No
Binary and oversized-file policy Yes No No No
Typed skip reasons and warnings Yes No No No
Symlinks skipped by default / loop detection Yes Yes Yes Configurable
Parallel collected / streaming traversal Yes / visitor + bounded pull No / callback No / serial iterator No / ordered iterator
Deterministic backpressured scan sink Yes No No No
Parallel one-pass verified content callback Yes No No No
Multi-root verified content callback Yes No No No
Changed-path-only content callback Yes No No No
Streaming content without retained manifest Yes No No No
Parallel pull iterator Bounded unordered / ordered DFS No (callback API) No Ordered DFS
Parallel multi-root raw traversal Collected + streaming Streaming callback No No
Parallel multi-root manifest scanner Yes No No No
Stateful per-directory batch Parallel ordered, typed No No Parallel ordered, typed
Redirected-stdout protection Yes Yes No No
Separate root-symlink policy Yes No Yes No
Cancellation and whole-scan budgets Yes Quit only No No
Minimum depth / hidden policy Yes / Yes Yes / Yes Yes / No Yes / Yes
Existing/dedicated worker pool Generic external / owned Internal threads Not applicable Rayon existing / new
Busy timeout / fallible submission External contract / Yes No / No Not applicable Yes / Yes
Measured 1M raw time / peak 264.7 ms / 7.6 MiB Not raw-equivalent 584.5 ms / 4.6 MiB 313.1 ms / 159.7 MiB
Measured 833k manifest time / peak Compact: 1,019.4 ms / 63.2 MiB 2,106.7 ms / 56.4 MiB Not a scanner Not a scanner
Default runtime dependencies 0 Unix / 1 Windows Multiple 2 platform helpers Rayon stack

Use Walker when you only need paths. Use Scanner when downstream results must be reproducible and explainable.

Remaining competitive boundaries

The functional gaps in retained-manifest memory, embeddable scheduling, parallel stateful batches, multi-root streaming, and traversal-free changed content are now closed. The remaining differences are evidence and ecosystem boundaries:

  1. Matcher production history. ignore remains the established Git-ignore implementation. Weavatrix checks representative, randomized, arbitrary-byte, and million-file exact-manifest parity, but does not claim equal ecosystem age.
  2. Cross-platform million-file evidence. CI and normal benchmarks cover Linux, Windows, and macOS, while the opt-in million-file RSS result below has so far been measured only on Windows.

The million-file result establishes top-tier performance on the measured Windows fixture, not a universal cross-platform ranking. The regular benchmark workflow still validates smaller output-equivalent corpora on Linux, Windows, and macOS; the million-file profile remains opt-in.

Install

[dependencies]
weavatrix-scan = "0.3"

Enable serialization only when needed:

[dependencies]
weavatrix-scan = { version = "0.3", features = ["serde"] }

Enable direct conversion from notify::Event without making a watcher runtime mandatory for other users:

weavatrix-scan = { version = "0.3", features = ["notify"] }

The default build has no third-party runtime dependency on Unix. Windows uses the small winapi-util safe wrapper for file, volume, and redirected-stdout identity because the equivalent std by-handle identity APIs remain unstable on the Rust 1.88 MSRV. Reimplementing that layer locally would require unsafe WinAPI FFI and still depend on Windows bindings; this crate keeps unsafe_code = "forbid".

Quick start

use weavatrix_scan::{ScanOptions, Scanner};

let options = ScanOptions::default()
    .with_extensions(["rs", "go", "ts", "py"])
    .with_parallelism(0);

let report = Scanner::new(".").options(options).scan()?;

println!("revision: {}", report.revision);
for file in &report.files {
    println!(
        "{}: {} bytes, hash={}",
        file.relative,
        file.bytes,
        file.content_hash.as_deref().unwrap_or("disabled")
    );
}
for skipped in &report.skipped {
    println!("skipped {}: {:?}", skipped.relative, skipped.kind);
}
# Ok::<(), weavatrix_scan::Error>(())

For the fastest path-only discovery, disable content reads:

use weavatrix_scan::{ScanOptions, Scanner};

let report = Scanner::new(".")
    .options(
        ScanOptions::default()
            .with_extensions(["rs", "go", "ts"])
            .metadata_only()
            .selected_files_only(),
    )
    .scan_compact()?;

assert!(report.files.iter().all(|file| file.content.is_none()));
# Ok::<(), weavatrix_scan::Error>(())

scan_compact discovers directly into root-shared records; it does not first build and convert a full report. Use report.absolute_path(file) only for the entries that need an owned absolute path. Rich compact scans retain hashes in optional boxed content evidence, accessible with file.content_hash().

Low-level walkers

Walker is a streaming iterator. It keeps paths as native PathBuf/OsStr values, uses iterative DFS, bounds open directory handles, and yields local errors according to policy:

use weavatrix_scan::{ErrorPolicy, WalkOptions, Walker};

let options = WalkOptions::default()
    .with_max_depth(Some(64))
    .with_max_open(8)
    .with_same_file_system(true)
    .with_error_policy(ErrorPolicy::Continue);

let mut walker = Walker::with_options(".", options)?;
while let Some(item) = walker.next() {
    match item {
        Ok(entry) => println!("{}", entry.path().display()),
        Err(error) => eprintln!("partial walk: {error}"),
    }
}
# Ok::<(), weavatrix_scan::WalkError>(())

The root is depth zero. After receiving a directory, callers can invoke skip_current_dir() before requesting the next item. Symbolic links are not followed by default; enabling .with_follow_links(true) keeps traversal inside the root and reports loops as typed skip reasons. The root itself follows a separate RootSymlinkPolicy: Follow is the compatibility default and Reject prevents an explicitly supplied symlink root from being opened.

WalkBuilder adds flexible low-level policies without changing the minimal streaming Walker:

use weavatrix_scan::WalkBuilder;

let entries = WalkBuilder::new("repo-a")
    .add_root("repo-b")
    .sort_by_file_name()
    .filter_directories(|entry| entry.file_name() != "target")
    .contents_first(true)
    .build()
    .collect::<Result<Vec<_>, _>>()?;
# Ok::<(), weavatrix_scan::WalkError>(())

sort_by receives complete std::fs::DirEntry values, including path, file type, and metadata access. sort_by_name remains the allocation-free native OsStr comparator, so sorting never requires lossy UTF-8 conversion. Low-level walkers accept either a directory or one file as the root. skip_stdout(true) prevents a redirected output file inside the tree from feeding back into a command that is scanning it. ParallelWalker applies the same option to collected, visitor, unordered-pull, and ordered-pull traversal; ParallelMultiWalker applies it to every root. Directory filters run before descent. filter_directories_stateful accepts FnMut, serializes callback access, and keeps one captured state across every root in the builder. For batch mutation and typed state propagation, StatefulWalkBuilder<R, E>::process_read_dir receives all immediate children before they are yielded. It can reorder or retain the batch, mutate R inherited by child directories, attach E to entries, and disable descent per entry.

use weavatrix_scan::StatefulWalkBuilder;

let entries = StatefulWalkBuilder::<usize, usize>::new(".", 0)
    .with_parallelism(0)
    .process_read_dir(|_, _, depth_state, entries| {
        *depth_state += 1;
        for entry in entries.iter_mut().filter_map(|item| item.as_mut().ok()) {
            entry.state = *depth_state;
        }
    })
    .build_parallel_ordered(1024)?
    .collect::<Result<Vec<_>, _>>()?;
# Ok::<(), weavatrix_scan::WalkError>(())

The parallel form runs each complete directory batch on the configured runtime, propagates callback-mutated state to child tasks, and yields strict DFS order under bounded backpressure. build() remains the zero-coordinator serial iterator.

ParallelWalker adapts between low-overhead frontier lanes and dynamic scheduling below narrow top-level trees:

use weavatrix_scan::ParallelWalker;

let report = ParallelWalker::new(".")
    .with_parallelism(0)
    .walk()?;
println!("entries={}, local_errors={}", report.entries.len(), report.errors.len());
# Ok::<(), weavatrix_scan::WalkError>(())

For pipelines that should parse entries immediately instead of collecting them, visit invokes a thread-safe callback directly on traversal workers:

use weavatrix_scan::{ParallelWalker, WalkControl, WalkEvent};

let summary = ParallelWalker::new(".").visit(|event| match event {
    WalkEvent::Entry(entry) if entry.file_name() == "target" => WalkControl::Skip,
    WalkEvent::Entry(entry) => {
        println!("{}", entry.path().display());
        WalkControl::Continue
    }
    WalkEvent::Error(error) => {
        eprintln!("{error}");
        WalkControl::Continue
    }
})?;
# Ok::<(), weavatrix_scan::WalkError>(())

Applications can isolate scans or reuse their own scheduler:

use weavatrix_scan::{ParallelRuntime, ParallelWalker, Scanner};

let runtime = ParallelRuntime::dedicated(8)?;
let raw = ParallelWalker::new(".")
    .runtime(runtime.clone())
    .walk()?;
let manifest = Scanner::new(".")
    .runtime(runtime)
    .scan_compact()?;

assert!(!raw.entries.is_empty());
println!("selected={}", manifest.files.len());
# Ok::<(), Box<dyn std::error::Error>>(())

ParallelRuntime::external accepts an Arc<dyn ParallelExecutor>. The executor receives each boxed job and the optional busy timeout, and can reject submission with io::Error; traversal reports that as WalkOperation::ScheduleWorker without waiting for an unsubmitted worker.

Consumers that prefer pull semantics can use a bounded iterator. A full buffer applies backpressure to traversal workers, and dropping the iterator cancels and joins its coordinator:

use weavatrix_scan::ParallelWalker;

for entry in ParallelWalker::new(".").into_iter_bounded(1024) {
    println!("{}", entry?.path().display());
}
# Ok::<(), weavatrix_scan::WalkError>(())

Use into_iter_ordered_bounded when consumers require strict deterministic DFS ordering. It prefetches directory reads in parallel while preserving the configured output capacity and max_open bound. Both pull modes cancel and join their coordinator when dropped. try_into_iter_bounded and try_into_iter_ordered_bounded report coordinator thread creation failures instead of panicking before traversal starts.

Larger bounded buffers improve throughput without changing the memory bound. Very small capacities are useful when minimum buffered state matters more than raw traversal speed.

ParallelMultiWalker::visit applies the same direct callback contract across multiple roots. Callback order is intentionally concurrent, every event is tagged with its root insertion index, and the returned reports stay in root insertion order:

use weavatrix_scan::{
    ParallelMultiWalker, WalkControl, WalkEvent,
};

let summary = ParallelMultiWalker::new("repo-a")
    .add_root("repo-b")
    .with_root_parallelism(2)
    .visit(|event| match event.event {
        WalkEvent::Entry(entry) if entry.file_name() == "target" => {
            WalkControl::Skip
        }
        WalkEvent::Entry(_) | WalkEvent::Error(_) => WalkControl::Continue,
    })?;

println!("roots={}, visited={}", summary.len(), summary.visited());
# Ok::<(), weavatrix_scan::WalkError>(())

WalkControl::Quit cooperatively cancels every active root. The cancellable form shares one CancellationToken; skip_stdout, traversal limits, error policy, and the selected ParallelRuntime apply to every root.

Parallel callbacks may start another walk using the same runtime. Such reentrant walks fall back to the iterative serial engine instead of waiting on workers that they already occupy. A callback panic stops and wakes the dynamic scheduler, is resumed on the caller, and leaves global, dedicated, or external execution reusable.

Scan modes

The same scanner supports three useful cost levels:

Mode Configuration Reads content Skip evidence Hashes content
Rich manifest ScanOptions::default() Yes Complete Yes
Safe discovery hash_file_contents = false First 8 KiB Complete No
Metadata only .metadata_only() No Complete No
Selected manifest .metadata_only().selected_files_only() No Omitted No

Traversal and content inspection use bounded available parallelism by default. Set .with_parallelism(1) for a serial run or pass a fixed worker count when a host application owns the wider scheduling policy. Use .with_traversal_parallelism(...) and .with_content_parallelism(...) when directory latency and content hashing need separate budgets.

For many independent roots, MultiScanner scans roots concurrently while returning reports in insertion order:

use weavatrix_scan::{MultiScanner, ScanOptions};

let reports = MultiScanner::new("workspace-a")
    .add_root("workspace-b")
    .options(ScanOptions::default().with_extensions(["rs", "go", "ts"]))
    .with_root_parallelism(2)
    .scan()?;

assert_eq!(reports.len(), 2);
# Ok::<(), weavatrix_scan::Error>(())

Scanner::scan_into keeps deterministic discovery metadata, then inspects and hands off one selected file at a time. The synchronous sink provides backpressure without an unbounded channel, and selected records are not retained by ScanStreamReport:

use weavatrix_scan::{ScanSinkControl, Scanner};

let summary = Scanner::new(".").scan_into(|file: &weavatrix_scan::ScannedFile| {
    println!("{}", file.relative);
    ScanSinkControl::Continue
})?;

assert_eq!(summary.selected, summary.emitted);
# Ok::<(), weavatrix_scan::Error>(())

For Search, Clone, and language adapters that need bytes, visit_content connects ignore-aware traversal to bounded parallel content workers. A worker-local callback receives borrowed chunks from the same read used for binary detection and optional SHA-256 evidence:

use weavatrix_scan::{
    ContentFileStatus, ContentValidationPolicy, ContentVisitControl,
    ContentVisitEvent, ScanOptions, Scanner,
};

let options = ScanOptions::default()
    .with_extensions(["rs", "go", "ts", "py"])
    .with_content_validation(ContentValidationPolicy::Strict);

let summary = Scanner::new(".")
    .options(options)
    .visit_content(|_worker_index| {
        let mut file_bytes = 0_u64;
        move |event| {
            match event {
                ContentVisitEvent::FileStart { .. } => file_bytes = 0,
                ContentVisitEvent::Chunk { bytes, .. } => {
                    file_bytes += u64::try_from(bytes.len()).unwrap();
                }
                ContentVisitEvent::FileEnd {
                    status: ContentFileStatus::Selected,
                    ..
                } => {
                    std::hint::black_box(file_bytes);
                }
                ContentVisitEvent::FileEnd { .. } => {}
            }
            ContentVisitControl::Continue
        }
    })?;

println!("files={}, bytes={}", summary.completed, summary.bytes_read);
# Ok::<(), weavatrix_scan::Error>(())

ContentVisitControl::SkipFile suppresses remaining chunks for one file while allowing required hash/binary evidence to finish. Quit cancels every worker. Events include root_index, root path, normalized relative path, and a monotonic sequence. Sort durable results by (root_index, relative). ContentValidationPolicy::Strict verifies native file evidence before and after the read; Fast keeps the safe opened-handle check but omits the post-read check for latency-sensitive local search. A deterministic total-byte budget automatically uses the compact two-phase path so budget selection remains path-order stable.

visit_content_streaming keeps the same byte, validation, cancellation, hash, and binary contracts but does not retain compact selected-file evidence or compute a revision. With selected_files_only() it has memory bounded by the queue, worker state, and one 64 KiB buffer per worker rather than selected-file count. ContentVisitReport::mode makes the empty streaming revision explicit. A deterministic max_total_bytes budget still requires the two-phase selection path.

MultiScanner::visit_content and visit_content_streaming use the selected runtime across all roots; the factory receives (root_index, worker_index) and reports remain in root insertion order. Scanner::visit_changed_content and visit_changed_content_streaming accept a safe file-only WatchPlan, read only existing changed paths, return removed paths separately, and yield FullRescanRequired before callbacks for structural or selection-changing plans.

Output contract

ScanReport contains:

  • root: canonical absolute repository root;
  • files: stable, lexicographically sorted ScannedFile values;
  • skipped: stable, sorted evidence for excluded entries;
  • warnings: non-fatal ignore-file and local I/O diagnostics;
  • ignore_sources: typed location and hash of every loaded selection input;
  • revision: SHA-256 digest over ignore inputs, selected paths, optional content hashes, portability, and partial-termination state;
  • complete: false when local errors made the evidence partial.
  • termination: typed reason for a bounded or cancelled partial scan;
  • portable: false when host-level Git configuration affected selection.
  • cache: content reads, strict-validation fingerprint reads, and strong hashes reused by an incremental scan.

Each ScannedFile contains an absolute path, slash-normalized repository path, byte size, optional sha256: content hash, whole-content cache fingerprint, and file-version evidence used to validate persistent cache reuse. The scanner compares size, timestamps, native file identity where available, and metadata before/after content reads. Native paths remain lossless in the walker and absolute PathBuf; invalid Unicode units in normalized manifest names are escaped (%XX on Unix, %uXXXX on Windows) instead of being replaced with the lossy Unicode replacement marker. With the serde feature, invalid native path units use a tagged byte/wide-unit representation and round-trip without loss; ordinary Unicode paths remain plain JSON strings.

Portable evidence and verified content

Use ScanReport::to_portable before sending scan evidence to another process, writing public logs, or attaching it to an AI request. PortableScanReport omits the absolute root, absolute file paths, file identities, timestamps, cache statistics, and free-form diagnostic text. Repository-relative paths and typed skip kinds remain; diagnostic details are represented only by stable SHA-256 values. External ignore-source locations are removed. Its selection_portable field separately records whether host-level Git configuration affected file selection.

Consumers reopening an existing snapshot can bind bytes back to the full local report:

use weavatrix_scan::{Scanner, SnapshotEvidence};

let report = Scanner::new(".").scan()?;
let portable = report.to_portable();
let content = report
    .content_provider()?
    .read_bounded("src/lib.rs", 2 * 1024 * 1024)?;

assert!(!portable.revision.is_empty());
assert_eq!(content.evidence, SnapshotEvidence::Sha256);
# Ok::<(), Box<dyn std::error::Error>>(())

SnapshotContentProvider accepts only sorted entries belonging to its report, rejects path escapes and symlinks, enforces an optional byte limit, and compares size plus native file-version evidence before and after reading. When a content hash exists, returned bytes must also match the recorded SHA-256. A missing or changed file produces typed SnapshotReadError::Stale evidence.

Weavatrix package boundary

Scan owns repository selection, safe bounded content delivery, hashes, revision, cancellation, and incremental deltas. Search owns literal/regex matching, line context, encodings, compressed inputs, indexes, and result formatting. Clone owns token normalization, Moss/winnowing, MinHash/LSH, Aho-Corasick, and clone grouping. Graph consumes normalized facts and keeps graph algorithms; content-search and clone algorithms do not belong there.

Incremental consumers

Two completed reports produce a stable changed-file set without filesystem access:

use weavatrix_scan::{DeltaQuality, Scanner};

let previous = Scanner::new(".").scan()?;
// Apply repository changes, then scan again.
let current = Scanner::new(".").scan_incremental(&previous)?;
let delta = current.delta_from(&previous);

assert!(matches!(
    delta.quality,
    DeltaQuality::ContentHash | DeltaQuality::Metadata | DeltaQuality::Partial
));
println!(
    "added={} modified={} removed={} renamed={}",
    delta.added.len(),
    delta.modified.len(),
    delta.removed.len(),
    delta.renamed.len()
);
# Ok::<(), weavatrix_scan::Error>(())

Unchanged files reuse prior SHA-256 values without reopening their content. Reports from another root, legacy reports without version evidence, and files whose size/version changed are read normally. Rename evidence is emitted only when the same content hash is unique in both manifests; duplicate-content moves remain explicit add/remove pairs instead of being guessed. Metadata-only deltas compare size plus available file-version evidence; callers that need content certainty should keep hashes enabled. Partial scans always produce DeltaQuality::Partial.

Persistent consumers should store ScanReport::to_cache() instead of the full report and pass it to Scanner::scan_cached. ScanCache has an explicit format version and contains only the canonical root plus relative path, size, version, hash, whole-content fingerprint, and binary-check evidence for reusable files. CacheValidationPolicy::Fast trusts matching file-version evidence. CacheValidationPolicy::Strict additionally reads a compact whole-content fingerprint before reusing the cached SHA-256, protecting coarse-timestamp and network filesystems from same-size, same-timestamp changes.

use weavatrix_scan::{CacheValidationPolicy, ScanOptions, Scanner};

let options = ScanOptions::default()
    .with_cache_validation(CacheValidationPolicy::Strict);
let first = Scanner::new(".").options(options.clone()).scan()?;
let second = Scanner::new(".").options(options).scan_cached(&first.to_cache())?;
assert_eq!(first.revision, second.revision);
# Ok::<(), weavatrix_scan::Error>(())

Long-lived file watchers can keep a RepositoryMatcher and call refresh() after an ignore input changes. Refresh builds a replacement matcher first, so a failure leaves the existing matcher usable.

WatcherEventAdapter converts create/modify/remove/rename notifications from any watcher library into sorted relative WatchPlan invalidations. Events outside the root are rejected, while directory, ignore-source, and explicit rescan events request a full scan. ScanCache::apply_watch_plan removes only affected entries or clears the cache when selection may have changed. Scanner::scan_watch_plan goes further: for a safe file-only plan it re-matches and inspects only changed paths, removes deleted paths from the previous manifest, keeps unchanged evidence, and recomputes the deterministic revision without traversing the tree. Structural, unsafe, partial, or selection-changing plans automatically use a complete scan. For indexes that consume bytes directly, visit_changed_content performs the same safe path matching and one-pass verified content delivery without walking unchanged directories. Its revision covers only the changed subset; visit_changed_content_streaming omits that subset manifest and revision. With the optional notify feature, plan_notify maps notify::Event batches directly. Access-only events are ignored; imprecise, rescan, and possibly structural events conservatively request a complete scan.

SkipKind distinguishes:

  • Binary
  • ConcurrentModification
  • Extension
  • FileSystemBoundary
  • Hidden
  • Ignored
  • IoError
  • MaxDepth
  • Override
  • Oversized
  • PathEscape
  • ScanLimit
  • StandardDirectory
  • Symlink
  • SymlinkLoop

This distinction matters to analyzers: "not selected by policy" is different from "unreadable" or "outside the repository."

Configuration

ScanOptions exposes:

Option Default Purpose
max_file_bytes 1,500,000 Reject oversized source candidates
extensions Empty Empty accepts every extension
file_types Empty Named file-name/repository-relative glob groups
ignore_files .gitignore, .ignore, .weavatrixignore Hierarchical local ignore files
ignore_policy Repository-only Optional parents, .git/info/exclude, global Git and explicit files
override_rules Empty Request-level include/exclude globs above ignore sources
ignore_case_insensitive false Optional ASCII case-insensitive ignore matching
skip_hidden false Skip dot-prefixed and Windows-hidden entries unless included
standard_skips Enabled Skip generated/vendor directories
hash_file_contents true Attach per-file hashes and content-sensitive revision
cache_validation Fast Trust file-version evidence, or verify a whole-content fingerprint in Strict mode
content_validation Strict Verify newly opened content before and after reading, or omit the post-read check in Fast mode
detect_binary_files true Reject files containing a NUL byte
evidence Complete Keep all typed exclusions, or only selected files
parallelism 0 Traversal/content workers; zero uses bounded available parallelism
traversal_parallelism None Optional traversal-only worker override
content_parallelism None Optional content-inspection worker override
limits.max_entries None Bound examined filesystem entries
limits.max_total_bytes None Deterministically bound selected content bytes
limits.timeout None Stop traversal/content inspection after a duration
cancellation None Cooperative cross-thread cancellation token
walk.max_depth None Limit entry depth; root is zero
walk.min_depth 0 Suppress shallower results while still traversing them
walk.max_open 64 Bound live directory handles/workers
walk.same_file_system false Stop at filesystem boundaries when enabled
walk.follow_links false Follow only in-root links and detect cycles
walk.root_symlink_policy Follow Follow or reject the explicitly supplied root symlink
walk.error_policy Continue Continue with partial typed evidence or abort
walk.collect_metadata true in ScanOptions Reuse directory-entry metadata without reopening selected paths

The standard directory policy skips:

.git .hg .svn .venv __pycache__ build coverage dist
node_modules target vendor

Disable it when another layer owns generated-directory policy:

use weavatrix_scan::{ScanOptions, StandardSkips};

let mut options = ScanOptions::default();
options.standard_skips = StandardSkips::Disabled;

NamedFileTypes::defaults() provides 265 deterministic language, markup, data, build, configuration, and infrastructure definitions backed by 678 patterns. That is a strict name-and-pattern superset of the 224 definitions and 594 patterns in ignore 0.4.31. len() and names() expose the catalog without activating it. Types can be composed, selected, and negated; later matching selections win:

use weavatrix_scan::NamedFileTypes;

let types = NamedFileTypes::defaults()
    .with_composed_type("product", ["rust", "go", "typescript", "javascript"])
    .select(["product"])
    .negate(["javascript"]);

Ignore semantics

Ignore files are loaded hierarchically with source precedence .weavatrixignore/custom > .ignore > .gitignore > .git/info/exclude > global Git. Deeper files win within the same source class. Supported Git-style constructs include:

  • comments and escaped leading # / !;
  • negation with !;
  • root-anchored patterns;
  • directory-only patterns;
  • *, **, and ?;
  • character classes, negated classes, and ranges;
  • brace alternatives such as {foo,bar};
  • escaped literals and escaped trailing spaces.

The default scanner intentionally does not read global Git configuration, parent rules outside the scan root, or .git/info/exclude; repository-local selection therefore stays portable. IgnorePolicy::git_compatible() enables all three explicitly inside Git repositories, records their content hashes, honors unconditional and matching includeIf Git config includes, and marks host-dependent reports non-portable. Local .gitignore, .ignore, and custom sources can be toggled independently. Request-level override globs use ignore::Override semantics: ordinary patterns include and leading ! patterns exclude. Explicit includes can opt paths back into standard-directory and extension filtering, but never bypass size or binary safety checks. RepositoryMatcher::matched exposes the winning typed decision without requiring a full walk. Differential tests compare exact selected path sets against the ignore crate for anchored, nested, negated, wildcard, and character-class fixtures plus 96-seed deterministic randomized rule sets and direct comparison with git check-ignore. A scheduled workflow also runs 100,000 arbitrary-byte grammar cases and deterministic directory-read fault injection. Stress cases cover deep trees, permission errors, raw non-UTF8 ignore rules/names, percent escapes, and followed symlink loops. The differential suite and competitor crates are dev-only.

Safety model

  • never executes repository code;
  • never starts subprocesses or accesses the network;
  • canonicalizes and validates the root before traversal;
  • does not follow symlink entries by default;
  • rejects followed links outside the canonical root and detects cycles;
  • follows in-root links in parallel using per-task ancestry, without a serial mode switch;
  • can enforce a same-filesystem boundary;
  • continues after independent local errors by default and marks the report partial;
  • caps selected file size before content reads;
  • rejects repository-local ignore-file symlinks and path traversal;
  • supports entry, total-byte, timeout, and cooperative cancellation bounds;
  • exports path-safe portable evidence without host paths or diagnostic text;
  • revalidates snapshot content before and after bounded consumer reads;
  • forbids unsafe Rust.

The scanner is read-only. Concurrent filesystem changes between discovery and the final metadata check are surfaced as ConcurrentModification warnings and skips under Continue, or as the first error under Abort.

Benchmarks

Run all included benchmarks:

cargo bench --locked

Run the competitor comparison:

cargo bench --locked --bench compare_competitors

The Competitor benchmarks workflow runs the same output-equivalent comparison on Ubuntu, Windows, and macOS for scanner or benchmark changes.

Run exact selected-path parity on a real repository:

$env:WEAVATRIX_BENCH_ROOT = "C:\path\to\repository"
cargo bench --locked --bench real_repository

Run skewed, deep, first-touch, bounded-handle, large-content, and incremental profiles:

cargo bench --locked --bench stress_profiles

Run root-policy, stateful-callback, bounded-pull, and watcher-adapter profiles:

cargo bench --locked --bench p2_apis

Create, verify, and measure an opt-in synthetic scale fixture outside the repository:

cargo bench --locked --bench scale_large -- prepare /tmp/weavatrix-scale 1000000
cargo bench --locked --bench scale_large -- verify /tmp/weavatrix-scale
cargo bench --locked --bench scale_large -- parallel-stream /tmp/weavatrix-scale 5
cargo bench --locked --bench scale_large -- scanner-compact /tmp/weavatrix-scale 5
cargo bench --locked --bench scale_large -- scanner /tmp/weavatrix-scale 5
cargo bench --locked --bench scale_large -- content-revision /tmp/weavatrix-scale 5
cargo bench --locked --bench scale_large -- content-stream /tmp/weavatrix-scale 5

The command refuses to populate an existing unmarked directory. The fixture uses 500 empty .rs files per directory; one sixth of its directories are excluded by a root .ignore. verify asserts the exact sorted path/size manifest from both full and compact scanners against ignore, not only the selected count. The profile is intentionally opt-in because creating and removing hundreds of thousands or millions of filesystem entries is itself expensive.

The synthetic comparison uses 6,000 source files across Rust, Go, and TypeScript in 80 sibling directories. It runs two warmups and 11 interleaved measured samples, then reports the median. Raw walkers must produce the same fully sorted native relative-path set; the ignore-aware comparison additionally checks the same normalized path-and-size manifest.

Sample result on Windows 11, Rust 1.97.1, warm filesystem cache, measured 2026-07-24 against ignore 0.4.31, walkdir 2.5.0, and jwalk 0.8.1:

Mode Library Files Median
Raw paths weavatrix Walker 6,004 29.0 ms
Raw paths weavatrix ParallelWalker 6,004 14.5 ms
Raw paths ignore 6,004 34.7 ms
Raw paths walkdir 6,004 31.9 ms
Raw paths jwalk 6,004 20.2 ms
Ignore-aware manifest weavatrix Scanner serial 6,001 55.8 ms
Ignore-aware manifest weavatrix Scanner parallel 6,001 32.9 ms
Ignore-aware manifest ignore 6,001 62.6 ms
Rich SHA-256 manifest weavatrix Scanner 6,000 237.1 ms

Each row is the median of five independent process medians. Every process runs 11 interleaved output-equivalent samples after two warmups. On this measurement ParallelWalker was 28.1% faster than jwalk; the parallel selected-manifest Scanner was 47.5% faster than ignore. The rich row additionally reads content, detects binaries, computes SHA-256 hashes, captures snapshot evidence, and records typed exclusions. Absolute timings vary by filesystem, cache, antivirus, CPU, and operating system; the benchmark workflow reruns the same checks on Ubuntu, Windows, and macOS.

The separate scale profile was measured on the same Windows host with a warm filesystem cache. Raw streaming rows are the median of seven independent process medians with seven measured runs after warmup. Metadata rows use five process medians with five runs, and the rich SHA-256 row uses three process medians with three runs. Peak working set is sampled in a fresh process containing one warmup and one measured run.

Work Implementation Files Median Peak
Raw streaming count ParallelWalker::visit 300,000 90.7 ms 6.9 MiB
Raw streaming count jwalk 300,000 96.6 ms 56.5 MiB
Raw collected paths ParallelWalker::walk 300,000 208.8 ms 162.5 MiB
Ignore-aware path/size manifest Scanner metadata-only 250,000 346.9 ms 97.3 MiB
Ignore-aware path/size manifest ignore 250,000 432.4 ms 20.5 MiB
No-ignore metadata manifest Scanner metadata-only 300,000 368.9 ms 157.5 MiB
Ignore-aware content manifest Scanner SHA-256 250,000 5,693.0 ms 235.9 MiB
Ignore-aware path emission rg --files to null 250,000 1,117.7 ms 35.3 MiB

The scanner row is a stronger contract than the comparison manifest: it also captures native version evidence, hashes ignore inputs, normalizes paths, sorts deterministically, and computes a revision. The ripgrep row is a whole-command throughput guardrail, not a library-equivalent benchmark: ripgrep formats and writes every path, while the library rows count or retain typed entries in-process. Content hashing is necessarily compared separately because rg --files does not open and hash every selected file. On this sample the bounded streaming walker stayed below jwalk in both median and peak memory, and metadata scanning stayed below both the output-equivalent ignore manifest and the ripgrep guardrail.

The same profile was then expanded to 1,000,000 files in 2,000 directories; 833,000 files passed .ignore. Raw rows below are the median of five independent process medians with five measured runs after warmup. The new compact/full/ignore manifest rows are the median of three independent process medians, each with one warmup and five measured runs; peak is the median of the three sampled process peaks. Serial, collected, no-ignore, rich, and ripgrep rows retain their earlier methodology described in the preceding revision of this benchmark.

Work Implementation Files Median Peak
Raw serial count Walker 1,000,000 546.3 ms 5.0 MiB
Raw streaming count ParallelWalker::visit 1,000,000 264.7 ms 7.6 MiB
Raw streaming count jwalk 1,000,000 313.1 ms 159.7 MiB
Raw serial count walkdir 1,000,000 584.5 ms 4.6 MiB
Raw collected paths ParallelWalker::walk 1,000,000 650.2 ms 529.0 MiB
Ignore-aware compact path/size manifest Scanner::scan_compact metadata-only 833,000 1,019.4 ms 63.2 MiB
Ignore-aware full path/size manifest Scanner::scan metadata-only 833,000 1,665.9 ms 309.6 MiB
Ignore-aware path/size manifest ignore 833,000 2,106.7 ms 56.4 MiB
No-ignore metadata manifest Scanner metadata-only 1,000,000 1,125.2 ms 369.6 MiB
Ignore-aware content manifest Scanner SHA-256 833,000 24,716.9 ms 776.2 MiB
Ignore-aware path emission rg --files to null 833,000 2,156.3 ms 42.3 MiB
No-ignore path emission rg --no-ignore --files to null 1,000,000 2,535.6 ms 95.1 MiB

On this million-file sample, streaming traversal was 15.5% faster than jwalk and used 95.2% less peak working set. The compact metadata Scanner was 51.6% faster than the output-equivalent ignore manifest while using 6.8 MiB more peak working set. Compared with the compatibility-oriented full report, it reduced peak memory by 79.6% and median time by 38.8%. Use the full report when every entry needs an owned absolute path and file-version evidence; use the compact report for large retained manifests, streaming traversal for raw consumers, and rich hashing only when content evidence is required.

The API benchmark uses the same corpus and methodology. Five process medians measured the ordered bounded DFS iterator at 6.4 ms versus 8.2 ms for jwalk. The new parallel typed stateful batch iterator measured 3.4 ms versus 4.3 ms for jwalk::process_read_dir, 22.2% faster on this sample while preserving the same batch mutation, child-state propagation, pruning, and ordered output contract. Output-equivalent streaming over two roots and 12,008 files measured 3.375 ms for Weavatrix versus 7.408 ms for ignore::build_parallel, 54.4% faster on this sample. Each process result is itself the median of 11 interleaved measured runs after two warmups, not a single best run. Watcher planning and changed-path scan profiles remain in the same reproducible benchmark.

The one-pass content profile selects and reads the same 6,001 tiny files in every case. Fast and the corresponding ignore baseline both validate the opened handle once; Strict and its baseline validate before and after the read. The raw ignore row intentionally omits snapshot validation and is the lower-contract throughput floor:

Content pipeline Retention / validation Median
Weavatrix visit_content_streaming None / opened handle 62.793 ms
Weavatrix visit_content_streaming None / before and after 73.872 ms
Weavatrix visit_content Revision / opened handle 83.307 ms
Weavatrix visit_content Revision / before and after 78.475 ms
ignore + File::read None / opened handle 76.139 ms
ignore + File::read None / before and after 80.529 ms
ignore + File::read None / unchecked 70.748 ms

These are five independent process medians from 11 interleaved samples after two warmups, measured on the Windows host above. Streaming Fast was 11.2% faster than unchecked ignore despite validating the opened handle; Streaming Strict was 8.3% faster than the equivalent before/after baseline. The two-root streaming profile processed 12,002 files in 152.013 ms versus 163.588 ms for verified ignore, 7.1% faster. A safe 1,024-file changed plan completed in 47.879 ms versus 109.694 ms for a full 6,000-file scan.

The scale memory check used a temporary 100,000-file fixture with 83,000 selected empty files. A fresh-process sample measured 7.3 MiB peak for streaming versus 39.4 MiB for revision retention, an 81.5% reduction; three-run medians were 2,109.956 and 2,182.466 ms respectively. This validates the scanner-to-consumer handoff, not future literal or regex matching: final comparison with ripgrep belongs to the Search package and must include matched output, line handling, and encoding policy.

Source review explains the remaining differences:

  • walkdir streams unsorted directory entries and bounds open descriptors;
  • jwalk schedules read_dir work through Rayon and restores ordered output;
  • ignore compiles patterns into GlobSet matchers and shares inherited matchers;
  • Weavatrix Walker streams iterative DFS, bounds live handles and buffers the oldest remaining frame only when max_open is reached; its plain-entry fast path and consuming WalkEntry::into_path avoid universal-policy checks and long-path clones in raw traversal;
  • Weavatrix ParallelWalker expands a small shallow frontier for narrow roots, then uses up to 16 Windows or 8 Unix workers without serially over-expanding small trees; bounded lanes keep report order independent of worker completion;
  • Weavatrix Scanner reuses inherited rules, indexes exact literals, specializes prefix/suffix globs, prefilters complex patterns, and sorts only the final report.

The optional real-repository benchmark keeps repository identities, paths, and individual measurements local. Published documentation contains only synthetic and hosted-runner corpus results. Every local comparison first asserts the exact same sorted (normalized path, bytes) manifest.

The synthetic stress profile measured a skewed raw tree at 6.7 ms (ParallelWalker), 8.2 ms (jwalk), and 14.1 ms (walkdir). The expanded synthetic deep-tree profile contains 60 levels and 7,680 files; five independent process medians measured 18.9 ms for Walker and 20.0 ms for walkdir, making Walker 5.6% faster on that sample. An unchanged synthetic 12 MiB SHA-256 manifest fell from 164.4 ms full scan to 1.9 ms with incremental hash reuse. Treat these as reproducible samples, not universal constants.

Correctness checks

The test suite covers:

  • deterministic results and revisions;
  • ignore-rule precedence and nested ignore files;
  • repository-only, Git-exclude, parent, explicit and reusable-matcher policies;
  • representative and randomized parity with ignore;
  • raw entry parity with walkdir and jwalk;
  • opt-in exact path/size manifest parity with ignore at arbitrary scale, including the measured 1,000,000-file fixture;
  • iterative deep trees, bounded handles, local error continuation, non-UTF8 paths, and symlink loops;
  • concurrent mutation detection and same-size incremental changes;
  • strict cache validation under simulated size/timestamp collisions;
  • multi-root walking, named file types, custom native sorting, directory filtering, and contents-first ordering;
  • single-file roots, full-entry sorting, built-in type composition/negation, strict catalog superset parity with ignore, redirected-stdout protection, parallel raw roots, and notify conversion;
  • binary, oversized, extension, generated-directory, and symlink policies;
  • serial/parallel content-inspection equivalence;
  • streaming parallel pruning and cancellation;
  • ordered bounded parallel DFS and parallel followed-link cycle handling;
  • global, dedicated, and rejecting external runtimes, typed submission failures, reentrant callbacks, panic propagation, and pool reuse;
  • parallel multi-root callback tagging, subtree pruning, global quit, cancellation, and deterministic per-root reports;
  • serial/parallel stateful batch order, pruning, entry state, inherited child state, and multi-worker execution;
  • full/compact exact manifest and revision equivalence;
  • changed-path watcher manifests, arbitrary-byte ignore grammar, and injected directory-read failures;
  • manifest delta evidence and live matcher refresh;
  • optional Serde support.

The real-repository benchmark compares the complete normalized selected-path set against ignore. Its comparison policy disables Weavatrix's file-size cap so an oversized file cannot masquerade as an ignore-rule mismatch.

Development

cargo fmt --all -- --check
cargo test --locked --all-features
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo doc --locked --no-deps --all-features
cargo bench --locked
cargo publish --locked --dry-run

The MSRV is Rust 1.88. CI checks Rust 1.88 on Linux, Windows, and macOS, with stable test coverage on all three platforms.

Relationship to Weavatrix

weavatrix-scan owns repository discovery. It does not parse languages or build graphs. weavatrix-graph owns typed graph primitives. Higher-level Weavatrix crates can compose both without coupling either library to MCP, a CLI, or language-specific parsers.

License

MIT © 2026 Sergii Ziborov.