srusty-files 0.2.0

Overview

Rusty Files is a production-grade file indexing and search library designed to be embedded in file managers, IDEs, system utilities, and any application requiring fast file discovery capabilities. It provides:

• Lightning-fast search across millions of files
• Multiple search modes: exact, fuzzy, regex, and glob patterns
• Rich filtering: by extension, size, date, and custom patterns
• Content search: full-text search within files
• Real-time updates: file system watching with automatic index synchronization
• Cross-platform: works on Windows, Linux, and macOS
• Persistent indexing: SQLite-based storage for instant startup
• Simple API: easy to integrate into any Rust project

Features

Core Capabilities

• ✅ Fast Indexing: Multi-threaded directory traversal (10,000+ files/second on SSD)
• ✅ Multiple Search Modes: Exact, case-insensitive, fuzzy, regex, and glob matching
• ✅ Advanced Filtering: Filter by extension, size, modification date
• ✅ Content Search: Full-text search in text files with encoding detection
• ✅ File System Watching: Real-time index updates on file changes
• ✅ Smart Ranking: Results ranked by relevance, recency, and path depth
• ✅ Incremental Updates: Only index changed files
• ✅ Exclusion Rules: Support for .gitignore patterns and custom rules
• ✅ Persistent Index: SQLite-based storage with automatic migrations
• ✅ Thread-safe: Concurrent indexing and searching
• ✅ CLI Application: Full-featured command-line interface

Installation

As a Library

Add to your Cargo.toml:

[dependencies]
srusty-files = "0.1"

As a CLI Tool

cargo install srusty-files

Or build from source:

git clone https://github.com/Aryagorjipour/rusty-files.git
cd rusty-files
cargo build --release

The binary will be available at target/release/filesearch.

Quick Start

Library Usage

use rusty_files::prelude::*;

fn main() -> Result<()> {
    let engine = SearchEngine::new("./index.db")?;

    engine.index_directory("/path/to/search", None)?;

    let results = engine.search("*.rs")?;

    for result in results {
        println!("{}", result.file.path.display());
    }

    Ok(())
}

CLI Usage

filesearch index /path/to/directory

filesearch search "*.rs"

filesearch search "test ext:rs size:>1KB modified:today"

filesearch interactive

filesearch stats

Documentation

Library API

Creating a Search Engine

use rusty_files::prelude::*;

let engine = SearchEngine::builder()
    .index_path("./index.db")
    .thread_count(8)
    .enable_content_search(true)
    .enable_fuzzy_search(true)
    .cache_size(1000)
    .exclusion_patterns(vec![
        ".git".to_string(),
        "node_modules".to_string(),
        "target".to_string(),
    ])
    .build()?;

Indexing Directories

let count = engine.index_directory("/path/to/dir", Some(Box::new(|progress| {
    println!("Indexed: {}/{}", progress.current, progress.total);
})))?;

println!("Indexed {} files", count);

Searching

Simple search:

let results = engine.search("filename")?;

Advanced queries:

use rusty_files::{Query, MatchMode, SizeFilter};

let query = Query::new("test".to_string())
    .with_match_mode(MatchMode::Fuzzy)
    .with_extensions(vec!["rs".to_string(), "toml".to_string()])
    .with_size_filter(SizeFilter::GreaterThan(1024))
    .with_max_results(100);

let results = engine.search_with_query(&query)?;

Query string format:

let results = engine.search("pattern ext:rs,txt size:>1MB modified:today mode:fuzzy")?;

File System Watching

engine.start_watching("/path/to/watch")?;

std::thread::park();

engine.stop_watching()?;

Incremental Updates

let stats = engine.update_index("/path/to/dir", None)?;

println!("Added: {}, Updated: {}, Removed: {}",
    stats.added, stats.updated, stats.removed);

Index Management

let stats = engine.get_stats()?;
println!("Total files: {}", stats.total_files);
println!("Index size: {}", stats.index_size);

let verification = engine.verify_index("/path/to/dir")?;
println!("Health: {:.1}%", verification.health_percentage());

engine.vacuum()?;

engine.clear_index()?;

Query Syntax

The query parser supports the following syntax:

• Basic search: filename
• Extension filter: pattern ext:rs or pattern ext:rs,txt,md
• Size filter:
  • pattern size:>1MB (greater than)
  • pattern size:<500KB (less than)
  • pattern size:1KB..10MB (range)
• Date filter:
  • pattern modified:today
  • pattern modified:yesterday
  • pattern modified:7days or pattern modified:1week
  • pattern modified:>2023-01-01
• Match mode: pattern mode:fuzzy, mode:regex, mode:glob, mode:exact
• Search scope: pattern scope:content, scope:path, scope:name
• Result limit: pattern limit:100

CLI Commands

Index Commands

filesearch index <path>
filesearch index /home/user/projects --progress

filesearch update <path>
filesearch update /home/user/projects --progress

Search Commands

filesearch search "query"

filesearch search "*.rs"

filesearch search "test ext:rs size:>1KB modified:today"

filesearch search "function mode:regex scope:content"

Management Commands

filesearch stats

filesearch verify <path>

filesearch watch <path>

filesearch clear --confirm

filesearch vacuum

Export

filesearch export --output results.json --query "*.rs"

filesearch export --output results.txt --query "test"

Interactive Mode

filesearch interactive

Interactive commands:

• :help - Show help
• :stats - Show index statistics
• :history - Show search history
• :clear - Clear screen
• :quit - Exit

Configuration

Configuration can be loaded from TOML or JSON files:

[config]
index_path = "./filesearch.db"
thread_count = 8
max_file_size_for_content = 10485760  # 10MB
enable_content_search = true
enable_fuzzy_search = true
fuzzy_threshold = 0.7
cache_size = 1000
bloom_filter_capacity = 10000000
bloom_filter_error_rate = 0.0001
max_search_results = 1000
batch_size = 1000
follow_symlinks = false
index_hidden_files = false
exclusion_patterns = [".git", "node_modules", "target", ".DS_Store"]
watch_debounce_ms = 500
enable_access_tracking = true
db_pool_size = 10

Load configuration:

use rusty_files::SearchConfig;

let config = SearchConfig::from_file(&PathBuf::from("config.toml"))?;
let engine = SearchEngine::with_config("./index.db", config)?;

Performance

Benchmarks

Benchmark results from actual runs on the test environment:

Indexing Performance

Files	Time (avg)	Throughput
100 files	101.96 ms	~981 files/sec
500 files	406.47 ms	~1,230 files/sec
1000 files	1.0491 s	~953 files/sec
Incremental update	646.82 ms	N/A

Search Performance

Operation	Time (avg)	Description
Simple search	1.47 ms	Basic filename matching
Pattern search	95.9 µs	Glob pattern matching
Fuzzy search	1.25 ms	Fuzzy matching algorithm
Filtered search	433.4 µs	Search with filters
Complex search	1.22 ms	Multi-criteria search

Test Environment: Linux 4.4.0, Release build with optimizations

Notes:

• Indexing performance scales well with parallel processing
• Search operations are sub-millisecond to low-millisecond range
• Pattern matching is extremely fast (~96 µs)
• Memory usage: <100MB base + configurable cache
• Startup time: <100ms with pre-built index

Optimization Tips

1. Adjust thread count: Set thread_count to CPU cores × 2
2. Tune cache size: Larger cache = faster repeated searches
3. Disable content search: If you don't need it, disable for faster indexing
4. Use exclusion patterns: Skip unnecessary directories
5. Batch operations: Use batch indexing for large directories

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     PUBLIC API LAYER                         │
│  ┌────────────┐  ┌──────────────┐  ┌──────────────────┐    │
│  │SearchEngine│  │ QueryBuilder │  │ ConfigManager    │    │
│  └────────────┘  └──────────────┘  └──────────────────┘    │
└────────────┬────────────────┬────────────────┬──────────────┘
             │                │                │
┌────────────▼────────────────▼────────────────▼──────────────┐
│                    CORE ENGINE LAYER                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │ Indexer  │  │ Searcher │  │ Watcher  │  │ Ranker   │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘   │
└────────────┬────────────┬────────────┬──────────────────────┘
             │            │            │
┌────────────▼────────────▼────────────▼──────────────────────┐
│                   DATA ACCESS LAYER                          │
│  ┌────────────┐  ┌──────────────┐  ┌──────────────────┐    │
│  │ Index DB   │  │ Cache Manager│  │ Bloom Filter     │    │
│  │ (SQLite)   │  │ (LRU)        │  │                  │    │
│  └────────────┘  └──────────────┘  └──────────────────┘    │
└──────────────────────────────────────────────────────────────┘

Examples

See the examples/ directory for complete examples:

• basic_search.rs - Simple file searching
• async_indexing.rs - Asynchronous indexing
• custom_config.rs - Custom configuration

Run examples:

cargo run --example basic_search

Testing

Run tests:

cargo test

cargo test --all-features

cargo test --release

Run benchmarks:

cargo bench

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Built with:

• tokio - Async runtime
• rusqlite - SQLite bindings
• walkdir - Directory traversal
• notify - File system watching
• fuzzy-matcher - Fuzzy matching
• clap - CLI parsing

Support

For issues, questions, or suggestions, please open an issue.