Crate fdf 

fdf - A High-Performance Parallel File System Traversal Library

fdf is a Rust library designed for efficient, parallel directory traversal with extensive filtering capabilities. It leverages Rayon for parallel processing and uses platform-specific optimisations for maximum performance.

This will be renamed before a 1.0!

Features

• Parallel Processing: Utilises Rayon’s work-stealing scheduler for concurrent directory traversal
• Platform Optimisations: Linux/Android specific getdents system calls for optimal performance, with fallbacks for other platforms
• Flexible Filtering: Support for multiple filtering criteria:
  • File name patterns (regex,glob)
  • File size ranges
  • File types (regular, directory, symlink, etc.)
  • File extensions
  • Hidden file handling
  • Custom filter functions
• Cycle Detection: Automatic symlink cycle prevention using inode caching
• Depth Control: Configurable maximum search depth
• Path Canonicalisation: Optional path resolution to absolute paths
• Error Handling: Configurable error reporting with detailed diagnostics

Performance Characteristics

• Uses mimalloc as global allocator on supported platforms for improved memory allocation performance
• Batched result delivery to minimise channel contention
• Zero-copy path handling where possible
• Avoids unnecessary stat calls through careful API design
• Makes up to 50% less getdents syscalls on linux/android, check the source code)

Platform Support

• Linux/Android: Optimised with direct getdents system calls
• BSD’s/macOS: Standard readdir with potential for future changes like fts_open (unlikely probably)
• Other Unix-like: Fallback to standard library functions
• Windows: Not currently supported (PRs welcome!)

Quick Start

use fdf::{walk::Finder, fs::DirEntry, SearchConfigError};
use std::sync::mpsc::Receiver;

fn find_files() -> Result<impl Iterator<Item = DirEntry>, SearchConfigError> {
   let finder = Finder::init("/path/to/search")
       .pattern("*.rs")
       .keep_hidden(false)
       .max_depth(Some(3))
       .follow_symlinks(true)
       .canonicalise_root(true) // Resolve the root to a full path
       .build()?;

   finder.traverse()
}

Setting custom filters example

use fdf::{walk::Finder, filters::FileTypeFilter};
use std::error::Error;

fn main() -> Result<(), Box<dyn Error>> {
    let finder = Finder::init("/var/log")
        .pattern("*.log")
        .keep_hidden(false)
        .type_filter(Some(FileTypeFilter::File))
        //set the custom filter
        .filter(Some(|entry| {
            entry.extension().is_some_and( |ext| ext.eq_ignore_ascii_case(b"log"))
        }))
        .build()?;

    let entries = finder.traverse()?;
    let mut count = 0;

    for entry in entries {
        count += 1;
        println!("Matched: {}", entry.as_path().display());
    }

    println!("Found {} log files", count);
    Ok(())
}

Re-exports

pub use chrono;

pub use libc;

Modules

filters

fs

util

walk

Macros

access_dirent

A helper macro to safely access dirent(64 on linux)’s fields of a libc::dirent/libc::dirent64 aka ‘dirent-type’ struct by offset.

const_assert

A compile time assert, mirroring static_assert from C++

const_from_env

Macro to create a const from an env var with compile-time parsing,

Structs

SearchConfig

This struct holds the configuration for searching a File system via traversal

TraversalError

Represents a traversal error encountered during directory processing

Enums

DirEntryError

Errors that can occur when reading or processing directory entries

FilesystemIOError

This enum encapsulates all possible I/O errors that can occur during filesystem operations,

SearchConfigError

Error type for search configuration and pattern compilation failures.