Crate ferralk 

Parallel filesystem walking with byte-first glob selection.

A Walker takes one or more roots, root-relative include and exclude patterns, and an explicit traversal policy, and returns entries as native Path values. The patterns prune the walk: {src,packages}/**/*.rs never opens a directory outside src and packages. Git ignore rules, symlink following, hidden-entry matching, metadata, and sorting stay off until asked for; recoverable errors are collected next to the entries.

use std::error::Error;

use ferralk::{ErrorPolicy, WalkOptions, Walker};

let result = Walker::new(".")
    .include("src/**/*.rs")?
    .exclude("**/generated/**")?
    .respect_git_ignore(true)
    .error_policy(ErrorPolicy::Collect)
    .options(WalkOptions::default().files_only(true).sort(true))
    .collect()?;

// `collect()?` succeeded, but that does not mean the walk did: under the
// default `ErrorPolicy::Collect` even a root that does not exist is reported
// in `errors()` below rather than as `Err`.
for entry in result.entries() {
    println!("{}", entry.path().display());
}
for error in result.errors() {
    // `Display` names the operation and path; `source()` says why it failed.
    match error.source() {
        Some(cause) => eprintln!("{error}: {cause}"),
        None => eprintln!("{error}"),
    }
}

options sets every WalkOptions switch in one call; see Walker::options. The table on ErrorPolicy shows which errors each policy collects, discards, or returns as Err.

Four ways to consume a walk:

• Walker::collect walks in parallel and returns every entry and every recoverable error in one WalkResult.
• Walker::stream walks on the calling thread and yields entries one at a time, in traversal order.
• Walker::stream_parallel walks in parallel on threads of its own and yields entries one at a time on the calling thread, in no particular order.
• Walker::visit runs a predicate on the worker that found each entry, so a caller-side filter does not become a serial pass afterwards.

Paths stay PathBuf throughout the public API. Patterns are matched against root-relative encoded path bytes; no filesystem result is converted through UTF-8. An entry’s path is its root joined with that relative path, so Walker::new(".") yields ./src/lib.rs, and relative_path is the src/lib.rs the patterns saw. ferralk_glob::path_bytes hands either to a ferralk_glob::Pattern of your own without allocating.

The usage guide lists every default and the switch that changes it. The stability contract states what 1.x promises.

Recipes

Every recipe is a self-contained function you can copy, followed by a call to it; the hidden lines only build a small tree to call it on. Matching paths you already hold, without a filesystem, is covered by the ferralk_glob recipes. The same recipes run as programs from crates/ferralk/examples, for example cargo run -p ferralk --example gitignore_walk -- <root>.

List files, respecting .gitignore

Git ignore rules are off until asked for. With them on, the .gitignore and .ignore files in the root and below it apply whether or not the root is inside a Git repository; inside one, .git/info/exclude and the files between the repository root and the walk root apply too. Walker::respect_git_ignore has the details. (The tree built for the example below is not a repository.)

use std::{
    error::Error,
    path::{Path, PathBuf},
};

use ferralk::{WalkOptions, Walker};

/// Every `.rs` file below `root` that the ignore rules leave in, relative
/// to `root` and sorted.
fn rust_files(root: &Path) -> Result<Vec<PathBuf>, Box<dyn Error + Send + Sync>> {
    let result = Walker::new(root)
        .include("**/*.rs")?
        .respect_git_ignore(true)
        .options(WalkOptions::default().files_only(true).sort(true))
        .collect()?;

    // `collect()` returned `Ok`, but a directory that could not be read is
    // only reported here.
    for error in result.errors() {
        eprintln!("not walked: {error}");
    }

    // `relative_path()` is `path()` without the root: the part the
    // patterns matched.
    let files = result.entries().iter().map(|entry| entry.relative_path().to_path_buf());
    Ok(files.collect())
}

assert_eq!(
    rust_files(&root)?,
    [Path::new("src/lib.rs"), Path::new("src/parser/mod.rs")],
);

Without an include, every entry that is not ignored is returned, directories and hidden files such as .gitignore itself included. WalkOptions::files_only drops directories, and WalkOptions::skip_hidden drops hidden entries the way the ignore crate does by default. sort(true) orders the entries as WalkOptions::sort describes.

Include and exclude lists

Includes are OR-ed, and an entry that any exclude matches is left out; a directory an exclude matches is not even opened. A fast-glob or globby list marks its excludes with a leading !, which the walker rejects rather than reading as negation, so sort the list into the two calls:

use std::{
    error::Error,
    path::{Path, PathBuf},
};

use ferralk::{WalkOptions, Walker};

/// The files below `root` that a fast-glob style list selects, relative to
/// `root` and sorted.
fn select(root: &Path, globs: &[&str]) -> Result<Vec<PathBuf>, Box<dyn Error + Send + Sync>> {
    let mut walker = Walker::new(root).options(WalkOptions::default().files_only(true).sort(true));
    for glob in globs {
        // `!(…)` is a negated extglob, not an exclude marker.
        match glob.strip_prefix('!').filter(|rest| !rest.starts_with('(')) {
            Some(excluded) => walker.try_exclude(excluded)?,
            None => walker.try_include(glob)?,
        };
    }
    let result = walker.collect()?;
    for error in result.errors() {
        eprintln!("not walked: {error}");
    }
    Ok(result.entries().iter().map(|entry| entry.relative_path().to_path_buf()).collect())
}

// For example from a configuration file.
let globs = ["src/**/*.ts", "!src/**/*.test.ts", "!**/generated/**"];
assert_eq!(select(&root, &globs)?, [Path::new("src/app/main.ts")]);

Three things differ from fast-glob. Patterns are anchored at the root, so target/** excludes only the top-level target and **/target/** every one; see Walker::exclude. Directories are returned unless WalkOptions::files_only says otherwise, where fast-glob defaults to onlyFiles: true. And try_include and try_exclude borrow the walker, so a caller can report an invalid pattern and go on with the rest instead of returning at the first one, as the include_exclude example does.

Stop early

Walker::stream yields entries on the calling thread as it finds them, so ordinary iterator adapters end the walk. Errors are items too; decide what to do with them before counting, as WalkStream explains. Inside a parallel walk, Verdict::Stop from a Walker::visit predicate ends it; from outside, a CancellationToken does (see the next recipes).

use std::{
    error::Error,
    ffi::OsStr,
    path::{Path, PathBuf},
    sync::OnceLock,
};

use ferralk::{Verdict, WalkEntry, Walker};

/// The first `count` `.rs` files directly in `root`, skipping errors.
fn first_rust_files(root: &Path, count: usize) -> Result<Vec<WalkEntry>, Box<dyn Error + Send + Sync>> {
    Ok(Walker::new(root)
        .include("*.rs")?
        .stream()
        .filter_map(Result::ok)
        .take(count)
        .collect())
}

/// Some `Cargo.toml` below `root`, ending the parallel walk at the first.
fn any_manifest(root: &Path) -> Result<Option<PathBuf>, Box<dyn Error + Send + Sync>> {
    let found = OnceLock::new();
    let result = Walker::new(root).visit(|entry| {
        if entry.basename() == Some(OsStr::new("Cargo.toml")) {
            let _ = found.set(entry.path().to_path_buf());
            Verdict::Stop
        } else {
            Verdict::Skip
        }
    })?;
    debug_assert_eq!(result.was_cancelled(), found.get().is_some());
    Ok(found.into_inner())
}

assert_eq!(first_rust_files(&root, 2)?.len(), 2);
assert!(any_manifest(&root)?.is_some());

Keep going when something cannot be read

The default ErrorPolicy::Collect walks on past a directory it cannot read and reports it in WalkResult::errors, a root that does not exist included, while collect() itself returns Ok. ErrorPolicy lists what Skip and Abort do instead. A WalkError’s Display names the operation and the path; the std::io::Error that says why is its source(), and WalkError::io_kind is that error’s kind.

use std::{error::Error, io, path::Path};

use ferralk::{WalkOperation, WalkResult, Walker};

/// Walks every root that can be read and reports the ones that cannot.
fn walk_all(roots: &[&Path]) -> Result<WalkResult, Box<dyn Error + Send + Sync>> {
    let (first, rest) = roots.split_first().ok_or("no roots given")?;
    let result = Walker::new(first).add_roots(rest.iter().copied())?.collect()?;
    for error in result.errors() {
        // Decide from the operation and the `io::ErrorKind`, never from the
        // message text.
        let cause = error.source().map(ToString::to_string).unwrap_or_default();
        match (error.operation(), error.io_kind()) {
            (WalkOperation::ReadDir, io::ErrorKind::NotFound) => {
                eprintln!("missing: {}", error.path().display());
            }
            _ => eprintln!("warning: {error}: {cause}"),
        }
    }
    Ok(result)
}

let missing = root.join("does-not-exist");
let result = walk_all(&[&root, &missing])?;
// The readable root was walked; the missing one is reported, not fatal.
assert_eq!(result.entries().len(), 2); // `src` and `src/lib.rs`
assert_eq!(result.errors().len(), 1);
assert_eq!(result.errors()[0].path(), missing);

Cancel a walk from another thread

A walk blocks its thread until it is done. Give it a CancellationToken and keep a clone: whoever holds the clone can stop it, and WalkResult::was_cancelled tells the walk’s owner that the entries are partial.

use std::{
    error::Error,
    path::PathBuf,
    thread::{self, JoinHandle},
};

use ferralk::{CancellationToken, WalkError, WalkResult, Walker};

/// Starts a walk on its own thread; the returned token stops it.
fn start_walk(root: PathBuf) -> (CancellationToken, JoinHandle<Result<WalkResult, WalkError>>) {
    let token = CancellationToken::default();
    let walker = Walker::new(root).cancellation(token.clone());
    (token, thread::spawn(move || walker.collect()))
}

let (token, walk) = start_walk(root.clone());
// Later, from a UI, a timeout, or a shutdown hook:
token.cancel();
let result = walk.join().expect("the walk thread does not panic")?;
if result.was_cancelled() {
    eprintln!("stopped early after {} entries", result.entries().len());
}

Walk from async code

There is no async API: a walk is blocking filesystem work, so run it on the runtime’s blocking pool. With Tokio, spawn_blocking does that, and a guard that cancels the token on drop stops the walk when the future is dropped, for example by a timeout or a losing select! branch. A Walker is Send, so it moves into the closure. (Not compiled here: this crate does not depend on Tokio.)

use std::{error::Error, path::PathBuf};

use ferralk::{CancellationToken, WalkResult, Walker};

/// Cancels the walk when the future that owns this guard is dropped.
struct CancelOnDrop(CancellationToken);

impl Drop for CancelOnDrop {
    fn drop(&mut self) {
        self.0.cancel();
    }
}

async fn rust_files(root: PathBuf) -> Result<WalkResult, Box<dyn Error + Send + Sync>> {
    let token = CancellationToken::default();
    let _cancel_on_drop = CancelOnDrop(token.clone());
    let walker = Walker::new(root)
        .include("**/*.rs")?
        .respect_git_ignore(true)
        .cancellation(token);
    let result = tokio::task::spawn_blocking(move || walker.collect()).await??;
    Ok(result)
}

Re-exports

pub use ferralk_glob;

Structs

CancellationToken

Cloneable, caller-controlled cooperative cancellation handle for a walk.

ParallelWalkStream

Incremental traversal produced by Walker::stream_parallel, walked on the configured workers and yielded on the thread that iterates it.

WalkEntry

One matching filesystem entry.

WalkError

A recoverable I/O failure observed while walking.

WalkOptions

Behavior switches for a Walker.

WalkResult

Completed entries and recoverable errors.

WalkResultIntoIter

Owning iterator over a WalkResult: every entry as Ok, then every error as Err.

WalkResultIter

Borrowing iterator over a WalkResult: every entry as Ok, then every error as Err.

WalkStream

Incremental portable traversal produced by Walker stream.

Walker

Builder for a filesystem walk: its roots, include and exclude patterns, and traversal policy.

Enums

ErrorPolicy

Controls what a walk does after a recoverable filesystem error.

Verdict

What a Walker::visit visitor decides about one entry.

WalkEntryKind

Filesystem kind observed for one walked entry.

WalkOperation

Filesystem operation associated with a recoverable walking error.

WildcardMode

How far an ordinary wildcard reaches in a walker pattern.

Constants

VERSION

Crate version exposed for build and integration diagnostics.