Expand description
§walkkit - Parallel filesystem walker with ignore-aware traversal
High-performance parallel directory walker with ignore-aware traversal, bounded work queues,
and cycle detection. walkkit provides both a custom multi-threaded engine (Walker) with
glob filtering and inode sorting, and a codebase scanner (CodeWalker) tuned for workspace
analysis with .gitignore hierarchy parsing and lazy content loading.
All filesystem errors are preserved explicitly as typed error items rather than silently swallowed. Binary files are detected via magic bytes and NUL-byte sampling, and symlink cycles are prevented using OS directory identifiers.
§Quick Start
use walkkit::{Walker, WalkItem};
let walker = Walker::new()
.add_root("./src")
.with_parallelism(4)
.respect_gitignore(true)
.skip_binary(true);
for item in walker.walk()? {
match item {
WalkItem::File(f) => println!("{}: {} bytes", f.path.display(), f.size),
WalkItem::Error(e) => eprintln!("Error: {}", e),
}
}§When to use / when not to use
When to use:
- Parallel directory traversal over large codebases or large nested directories.
- Codebase scanning requiring
.gitignorerules, glob include/exclude filters, and binary skipping. - Applications needing strict error reporting without silent loss during traversal.
When not to use:
- Simple single-directory listings where standard
std::fs::read_diris sufficient. - High-contention fine-grained parallel traversal across millions of tiny flat files where
jwalkwork-stealing is specifically needed.
§Compared to alternatives
walkkit provides structured parallel traversal with first-class WalkItem::Error propagation,
eliminating silent error suppression found in default iterators. Unlike walkdir (which is single-threaded),
walkkit leverages multi-threaded worker pools with bounded channels to bound in-flight memory.
Compared to ignore and jwalk, walkkit offers both a low-level bounded Walker and a high-level
CodeWalker abstraction, integrated with hashkit for content addressing and security probing.
§How it fits in Santh
walkkit lives in libs/performance/io as the foundational parallel directory discovery engine across
Santh analyzers, scanner tools, and indexing tools. It depends on hashkit for digest calculations
and provides the traversal primitive for codebase analysis pipelines.
Re-exports§
pub use codewalker::CodeWalker;pub use codewalker::FileContent;pub use codewalker::FileContentChunks;pub use codewalker::FileEntry;pub use codewalker::WalkConfig;pub use filter::CompiledFilter;pub use filter::FileFilter;
Modules§
- codewalker
- File tree walker - the core iteration engine.
- detect
- Binary file detection via magic bytes.
- error
- Error types for code tree walking and content loading.
- filter
- File filtering using include and exclude glob patterns.
- probe
- Shared file probing and enrichment primitives.
- sandbox
- Code protocol sandbox execution helpers.
Structs§
- Walk
Error - A filesystem error encountered during traversal (always reported, never swallowed).
- Walk
Item Iter - Iterator over walk results: discovered files and traversal errors.
- Walked
File - A discovered file with its path and essential metadata.
- Walker
- Builder and configuration for parallel file discovery.
Enums§
- Error
- Errors returned by walkkit operations.
- Sort
Mode - Specifies how the discovered files should be sorted. Note that sorting requires collecting all files in memory before yielding them.
- Walk
Item - One element from a directory walk: a file candidate or a traversal error.
- WalkOp
- What the walker was doing when an I/O error occurred.
Constants§
- MAX_
WALK_ PATH_ BYTES - Maximum byte length of any path we traverse or emit (bound TOCTOU / kernel limits).