Skip to main content

Crate walkkit

Crate walkkit 

Source
Expand description

§walkkit - Parallel filesystem walker with ignore-aware traversal

santh status

High-performance parallel directory walker with ignore-aware traversal, bounded work queues, and cycle detection. walkkit provides both a custom multi-threaded engine (Walker) with glob filtering and inode sorting, and a codebase scanner (CodeWalker) tuned for workspace analysis with .gitignore hierarchy parsing and lazy content loading.

All filesystem errors are preserved explicitly as typed error items rather than silently swallowed. Binary files are detected via magic bytes and NUL-byte sampling, and symlink cycles are prevented using OS directory identifiers.

§Quick Start

use walkkit::{Walker, WalkItem};

let walker = Walker::new()
    .add_root("./src")
    .with_parallelism(4)
    .respect_gitignore(true)
    .skip_binary(true);

for item in walker.walk()? {
    match item {
        WalkItem::File(f) => println!("{}: {} bytes", f.path.display(), f.size),
        WalkItem::Error(e) => eprintln!("Error: {}", e),
    }
}

§When to use / when not to use

When to use:

  • Parallel directory traversal over large codebases or large nested directories.
  • Codebase scanning requiring .gitignore rules, glob include/exclude filters, and binary skipping.
  • Applications needing strict error reporting without silent loss during traversal.

When not to use:

  • Simple single-directory listings where standard std::fs::read_dir is sufficient.
  • High-contention fine-grained parallel traversal across millions of tiny flat files where jwalk work-stealing is specifically needed.

§Compared to alternatives

walkkit provides structured parallel traversal with first-class WalkItem::Error propagation, eliminating silent error suppression found in default iterators. Unlike walkdir (which is single-threaded), walkkit leverages multi-threaded worker pools with bounded channels to bound in-flight memory.

Compared to ignore and jwalk, walkkit offers both a low-level bounded Walker and a high-level CodeWalker abstraction, integrated with hashkit for content addressing and security probing.

§How it fits in Santh

walkkit lives in libs/performance/io as the foundational parallel directory discovery engine across Santh analyzers, scanner tools, and indexing tools. It depends on hashkit for digest calculations and provides the traversal primitive for codebase analysis pipelines.

Re-exports§

pub use codewalker::CodeWalker;
pub use codewalker::FileContent;
pub use codewalker::FileContentChunks;
pub use codewalker::FileEntry;
pub use codewalker::WalkConfig;
pub use filter::CompiledFilter;
pub use filter::FileFilter;

Modules§

codewalker
File tree walker - the core iteration engine.
detect
Binary file detection via magic bytes.
error
Error types for code tree walking and content loading.
filter
File filtering using include and exclude glob patterns.
probe
Shared file probing and enrichment primitives.
sandbox
Code protocol sandbox execution helpers.

Structs§

WalkError
A filesystem error encountered during traversal (always reported, never swallowed).
WalkItemIter
Iterator over walk results: discovered files and traversal errors.
WalkedFile
A discovered file with its path and essential metadata.
Walker
Builder and configuration for parallel file discovery.

Enums§

Error
Errors returned by walkkit operations.
SortMode
Specifies how the discovered files should be sorted. Note that sorting requires collecting all files in memory before yielding them.
WalkItem
One element from a directory walk: a file candidate or a traversal error.
WalkOp
What the walker was doing when an I/O error occurred.

Constants§

MAX_WALK_PATH_BYTES
Maximum byte length of any path we traverse or emit (bound TOCTOU / kernel limits).