fsindex

Fast, powerful filesystem indexing with .gitignore support and an iterator-based API.

Features

Fast parallel traversal - Uses rayon for parallel file processing
.gitignore support - Respects .gitignore, .git/info/exclude, and global gitignore
Content hashing - XXH3 hashing for change detection
Language detection - Automatic programming language detection from file extensions
File watching - Real-time filesystem monitoring with notify
Flexible configuration - Builder pattern for easy customization
Iterator-based API - Memory efficient streaming over large directories

Installation

Add to your Cargo.toml:

[dependencies]
fsindex = "0.1"

Quick Start

use fsindex::FileIndexer;

let indexer = FileIndexer::new("./src");

for file in indexer.files() {
    println!("{}: {} bytes", file.path.display(), file.metadata.size);
}

Configuration

Use the builder pattern for custom configuration:

use fsindex::{FileIndexer, Config};

let config = Config::builder()
    .respect_gitignore(true)
    .include_hidden(false)
    .max_depth(Some(10))
    .extensions(vec!["rs", "toml"])
    .follow_symlinks(false)
    .parallel(true)
    .build();

let indexer = FileIndexer::with_config("./", config);

for file in indexer.files() {
    println!("{}", file.path.display());
}

Available Configuration Options

Option	Default	Description
`respect_gitignore`	`true`	Honor `.gitignore` files
`include_hidden`	`false`	Include hidden files/directories
`max_depth`	`None`	Maximum traversal depth (unlimited by default)
`extensions`	`[]`	Filter by file extensions (empty = all)
`exclude_patterns`	`[]`	Glob patterns to exclude
`include_patterns`	`[]`	Glob patterns to include
`follow_symlinks`	`false`	Follow symbolic links
`read_contents`	`true`	Read file contents into memory
`max_content_size`	`10MB`	Maximum file size for content reading
`custom_ignore_files`	`[]`	Additional ignore files
`parallel`	`true`	Use parallel traversal
`threads`	`0`	Thread count (0 = auto)

Parallel Processing

For large directories, use parallel file processing:

use fsindex::FileIndexer;

let indexer = FileIndexer::new("./");
let files = indexer.files_parallel(); // Collects and processes in parallel

for file in files {
    if let Some(hash) = file.hash_hex() {
        println!("{}: {}", file.path.display(), hash);
    }
}

File Watching

Monitor filesystem changes in real-time:

use fsindex::FileIndexer;

let indexer = FileIndexer::new("./src");
let watcher = indexer.watch().expect("Failed to create watcher");

for event in watcher.filtered_events() {
    match event {
        Ok(e) => println!("{:?}: {:?}", e.kind, e.path()),
        Err(e) => eprintln!("Error: {}", e),
    }
}

Language Detection

Files are automatically tagged with their detected programming language:

use fsindex::FileIndexer;

let indexer = FileIndexer::new("./src");

for file in indexer.files() {
    if let Some(lang) = file.metadata.language {
        println!("{}: {}", file.path.display(), lang);
    }
}

Supports 50+ languages including Rust, Python, JavaScript, TypeScript, Go, C/C++, Java, and many more.

Content Hashing

Each file includes an XXH3 hash for efficient change detection:

use fsindex::FileIndexer;

let indexer = FileIndexer::new("./");

for file in indexer.files() {
    if let Some(hash) = file.hash {
        println!("{}: {:016x}", file.path.display(), hash);
    }
}

Error Handling

Use files_result() for explicit error handling:

use fsindex::FileIndexer;

let indexer = FileIndexer::new("./");

for result in indexer.files_result() {
    match result {
        Ok(file) => println!("{}", file.path.display()),
        Err(e) => eprintln!("Error: {}", e),
    }
}

License

MIT

fsindex 0.1.0