walkkit 0.1.3

Parallel filesystem walker with ignore-aware traversal and bounded work queues
Documentation
# walkkit  -  Specification

## Overview

`walkkit` is a Rust filesystem-walking library that provides two traversal engines: a custom multi-threaded `Walker` with bounded work queues, glob filtering, and explicit cycle detection, and a higher-level `CodeWalker` (backed by the `ignore` crate) tuned for codebase scanning with TOML-configurable ignore rules, extension filtering, and lazy content loading. Both engines support symlink handling, binary-file skipping, depth limits, and streaming or sorted result modes.

## Architecture

- **`lib.rs`**  -  Public exports and shared types (`WalkItem`, `WalkError`, `WalkOp`, `Error`).
- **`walker.rs`**  -  `Walker` builder, `WalkedFile` metadata struct, and platform-specific `DirId` for cycle detection.
- **`worker.rs`**  -  Single-threaded (`walk_single_thread`) and multi-threaded (`walk_multi_thread`) traversal workers. The multi-threaded worker uses a `Mutex<WorkState>` + `Condvar` shared queue with an `active` counter to detect termination.
- **`walk_common.rs`**  -  Helpers for metadata resolution, sorted directory enumeration (`read_dir_sorted`), binary probing via NUL-byte sampling (with TOCTOU guards on Unix), and the `WalkOptions` struct.
- **`filter.rs`**  -  `FileFilter` / `CompiledFilter` using `globset`, plus `code_entry_allowed` and `code_process_path` for the code walker.
- **`iter.rs`**  -  `WalkItemIter` adapter that yields from either a live `crossbeam_channel::Receiver` or a pre-collected buffered vector.
- **`sort.rs`**  -  `SortMode` (`Unsorted`, `ByName`, `BySize`).
- **`codewalker/`**  -  `traverse.rs` (`CodeWalker` using `ignore::WalkBuilder`), `parallel.rs` (`walk_parallel` with `ignore::WalkParallel`), and `mod.rs` (`WalkConfig`, `FileEntry`, `FileContent`, `FileContentChunks`).
- **`detect.rs`**  -  Binary detection via magic-byte signatures and extension heuristics.
- **`archive/`**  -  Feature-gated (`archive`) ZIP/TAR parsing and gzip/zstd decompression helpers.
- **`sandbox.rs`**  -  Async script execution helpers for security scanning workflows.

Data flow: `Walker::walk()` compiles filters, spawns a control thread, which launches worker threads that push `WalkItem` values through a bounded channel to the consuming iterator.

## Guarantees

- **No silent failures**  -  Every I/O error becomes a `WalkItem::Error` with the path and operation class preserved.
- **Symlink cycle detection**  -  When `follow_symlinks` is enabled, directories are tracked by stable `(dev, ino)` identifiers (Unix) or canonicalized paths (fallback) to prevent infinite loops.
- **Duplicate suppression**  -  `visited_files` and `visited_dirs` sets prevent re-emitting the same filesystem entity within a single walk.
- **Path length bounding**  -  Paths exceeding `MAX_WALK_PATH_BYTES` (8192) are rejected with an explicit error instead of being silently skipped.
- **TOCTOU hardening**  -  On Unix, binary probing opens files with `O_NOFOLLOW` and verifies `ino`/`dev` match the prior `stat` before reading.

## Public API

### `Walker` (custom parallel walker)
- `Walker::new()`  -  Builder with default parallelism (4).
- `.add_root(path)`  -  Add a traversal root.
- `.with_parallelism(n)`  -  Set worker count (clamped to `[1, available_parallelism * 4]` capped at 256).
- `.with_filter(FileFilter)`  -  Glob include/exclude rules.
- `.with_sort(SortMode)`  -  `Unsorted`, `ByName`, or `BySize`.
- `.follow_symlinks(bool)`, `.respect_gitignore(bool)`, `.skip_binary(bool)`
- `.with_max_depth(d)`, `.with_extension_filter(ext)`, `.with_size_limit(bytes)`
- `.walk() -> Result<WalkItemIter, Error>`  -  Streaming or buffered iteration.
- `.try_walk_parallel() -> Result<Receiver<WalkItem>, Error>`  -  Direct channel access.

### `CodeWalker` (codebase scanner)
- `CodeWalker::new(root, WalkConfig)`  -  TOML-deserializable config with defaults for `skip_binary`, `skip_hidden`, `respect_gitignore`, `exclude_dirs`, `max_file_size`, etc.
- `.walk() -> Result<Vec<FileEntry>>`  -  Collect all entries.
- `.walk_sorted() -> Result<Vec<FileEntry>>`  -  Sorted by path.
- `.walk_iter() -> impl Iterator<Item = Result<FileEntry>>`  -  Lazy iteration.
- `.walk_parallel(threads) -> mpsc::Receiver<Result<FileEntry>>`  -  Parallel traversal via `ignore::WalkParallel`.

### Supporting types
- `WalkItem`  -  `File(WalkedFile)` or `Error(WalkError)`.
- `WalkedFile`  -  `{ path: PathBuf, size: u64, inode: u64 }`.
- `FileEntry`  -  `{ path: PathBuf, size: u64, is_binary: bool }` with `.content()`, `.content_chunks()`, `.content_str()`.
- `FileContent`  -  `Text(String)`, `Binary(Vec<u8>)`, or `Unknown(Vec<u8>)`.
- `FileFilter` / `CompiledFilter`  -  Builder + compiled glob matcher.

## Error handling

- **`WalkError`**  -  Non-fatal traversal errors (`Metadata`, `ReadDir`, `Open`, `Gitignore`). Always emitted as `WalkItem::Error`; the walk continues elsewhere when possible.
- **`Error`**  -  Fatal configuration errors from `Walker::walk()`: invalid glob syntax (`InvalidGlob`) or empty/NUL-containing filter patterns (`InvalidFilterPattern`).
- **`CodewalkError`**  -  Errors from `CodeWalker`: `Io`, `FileTooLarge(u64)`, `Ignore`, `Utf8Error`.

## Performance characteristics

- **Traversal**  -  `O(N)` in the number of directory entries visited; multi-threaded scaling is bounded by the coarse mutex-protected work queue (not work-stealing).
- **Memory**  -  Streaming mode (`SortMode::Unsorted`) holds only the bounded channel (1024 items) in flight. Sorted modes buffer all results, requiring `O(M)` memory for `M` matched files.
- **Binary detection**  -  First 8 KB read for files ≤ 10 MB; strided 64 KB samples for larger files. Magic-byte checks run before opening the file when the extension is in a known-binary list.
- **Directory enumeration**  -  Entries are sorted lexicographically by `OsStr` bytes to improve determinism; this adds `O(K log K)` per directory with `K` children.

## Limitations

- `Walker` does not use a work-stealing deque; high contention on the shared mutex can limit scaling on very large, shallow directories.
- `CodeWalker` relies on the `ignore` crate for traversal; its parallelism and gitignore semantics are inherited from that dependency.
- Binary detection is heuristic (magic bytes, extension list, and NUL-byte sampling) and may misclassify unusual text or binary formats.
- `Walker` deduplicates files by `PathBuf`, not by inode, so the same file reached via hard links under different names may be emitted multiple times.
- No async/`await` API for `Walker`; it spawns OS threads internally.
- `MAX_WALK_PATH_BYTES` is hard-coded to 8192; platforms with shorter `PATH_MAX` may still encounter OS-level errors before this bound is reached.
- The `archive` feature provides parsing helpers but does not integrate automatic archive expansion into the main `Walker` or `CodeWalker` traversal.