srusty-files 0.2.0

<p align="center">
  <img src="assets/RustyFiles-Logo.png" alt="Rusty Files Logo" width="300"/>
</p>

<h1 align="center">Rusty Files</h1>

<p align="center">
  A high-performance, cross-platform file search engine library and CLI written in Rust.
</p>

<p align="center">
  <a href="https://github.com/Aryagorjipour/rusty-files/actions"><img src="https://github.com/Aryagorjipour/rusty-files/workflows/Rust/badge.svg" alt="Build Status"></a>
  <a href="https://github.com/Aryagorjipour/rusty-files/releases"><img src="https://img.shields.io/github/v/release/Aryagorjipour/rusty-files" alt="Latest Release"></a>
  <a href="https://crates.io/crates/srusty-files"><img src="https://img.shields.io/crates/v/srusty-files" alt="Crates.io"></a>
  <a href="https://crates.io/crates/srusty-files"><img src="https://img.shields.io/crates/d/srusty-files" alt="Downloads"></a>
  <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow.svg" alt="License: MIT"></a>
  <a href="https://www.rust-lang.org"><img src="https://img.shields.io/badge/rust-1.70%2B-blue.svg" alt="Rust 1.70+"></a>
</p>

## Overview

Rusty Files is a production-grade file indexing and search library designed to be embedded in file managers, IDEs, system utilities, and any application requiring fast file discovery capabilities. It provides:

- **Lightning-fast search** across millions of files
- **Multiple search modes**: exact, fuzzy, regex, and glob patterns
- **Rich filtering**: by extension, size, date, and custom patterns
- **Content search**: full-text search within files
- **Real-time updates**: file system watching with automatic index synchronization
- **Cross-platform**: works on Windows, Linux, and macOS
- **Persistent indexing**: SQLite-based storage for instant startup
- **Simple API**: easy to integrate into any Rust project

## Features

### Core Capabilities

- ✅ **Fast Indexing**: Multi-threaded directory traversal (10,000+ files/second on SSD)
- ✅ **Multiple Search Modes**: Exact, case-insensitive, fuzzy, regex, and glob matching
- ✅ **Advanced Filtering**: Filter by extension, size, modification date
- ✅ **Content Search**: Full-text search in text files with encoding detection
- ✅ **File System Watching**: Real-time index updates on file changes
- ✅ **Smart Ranking**: Results ranked by relevance, recency, and path depth
- ✅ **Incremental Updates**: Only index changed files
- ✅ **Exclusion Rules**: Support for .gitignore patterns and custom rules
- ✅ **Persistent Index**: SQLite-based storage with automatic migrations
- ✅ **Thread-safe**: Concurrent indexing and searching
- ✅ **CLI Application**: Full-featured command-line interface

## Installation

### As a Library

Add to your `Cargo.toml`:

```toml
[dependencies]
srusty-files = "0.1"
```

### As a CLI Tool

```bash
cargo install srusty-files
```

Or build from source:

```bash
git clone https://github.com/Aryagorjipour/rusty-files.git
cd rusty-files
cargo build --release
```

The binary will be available at `target/release/filesearch`.

## Quick Start

### Library Usage

```rust
use rusty_files::prelude::*;

fn main() -> Result<()> {
    let engine = SearchEngine::new("./index.db")?;

    engine.index_directory("/path/to/search", None)?;

    let results = engine.search("*.rs")?;

    for result in results {
        println!("{}", result.file.path.display());
    }

    Ok(())
}
```

### CLI Usage

```bash
filesearch index /path/to/directory

filesearch search "*.rs"

filesearch search "test ext:rs size:>1KB modified:today"

filesearch interactive

filesearch stats
```

## Documentation

### Library API

#### Creating a Search Engine

```rust
use rusty_files::prelude::*;

let engine = SearchEngine::builder()
    .index_path("./index.db")
    .thread_count(8)
    .enable_content_search(true)
    .enable_fuzzy_search(true)
    .cache_size(1000)
    .exclusion_patterns(vec![
        ".git".to_string(),
        "node_modules".to_string(),
        "target".to_string(),
    ])
    .build()?;
```

#### Indexing Directories

```rust
let count = engine.index_directory("/path/to/dir", Some(Box::new(|progress| {
    println!("Indexed: {}/{}", progress.current, progress.total);
})))?;

println!("Indexed {} files", count);
```

#### Searching

Simple search:

```rust
let results = engine.search("filename")?;
```

Advanced queries:

```rust
use rusty_files::{Query, MatchMode, SizeFilter};

let query = Query::new("test".to_string())
    .with_match_mode(MatchMode::Fuzzy)
    .with_extensions(vec!["rs".to_string(), "toml".to_string()])
    .with_size_filter(SizeFilter::GreaterThan(1024))
    .with_max_results(100);

let results = engine.search_with_query(&query)?;
```

Query string format:

```rust
let results = engine.search("pattern ext:rs,txt size:>1MB modified:today mode:fuzzy")?;
```

#### File System Watching

```rust
engine.start_watching("/path/to/watch")?;

std::thread::park();

engine.stop_watching()?;
```

#### Incremental Updates

```rust
let stats = engine.update_index("/path/to/dir", None)?;

println!("Added: {}, Updated: {}, Removed: {}",
    stats.added, stats.updated, stats.removed);
```

#### Index Management

```rust
let stats = engine.get_stats()?;
println!("Total files: {}", stats.total_files);
println!("Index size: {}", stats.index_size);

let verification = engine.verify_index("/path/to/dir")?;
println!("Health: {:.1}%", verification.health_percentage());

engine.vacuum()?;

engine.clear_index()?;
```

### Query Syntax

The query parser supports the following syntax:

- **Basic search**: `filename`
- **Extension filter**: `pattern ext:rs` or `pattern ext:rs,txt,md`
- **Size filter**:
  - `pattern size:>1MB` (greater than)
  - `pattern size:<500KB` (less than)
  - `pattern size:1KB..10MB` (range)
- **Date filter**:
  - `pattern modified:today`
  - `pattern modified:yesterday`
  - `pattern modified:7days` or `pattern modified:1week`
  - `pattern modified:>2023-01-01`
- **Match mode**: `pattern mode:fuzzy`, `mode:regex`, `mode:glob`, `mode:exact`
- **Search scope**: `pattern scope:content`, `scope:path`, `scope:name`
- **Result limit**: `pattern limit:100`

### CLI Commands

#### Index Commands

```bash
filesearch index <path>
filesearch index /home/user/projects --progress

filesearch update <path>
filesearch update /home/user/projects --progress
```

#### Search Commands

```bash
filesearch search "query"

filesearch search "*.rs"

filesearch search "test ext:rs size:>1KB modified:today"

filesearch search "function mode:regex scope:content"
```

#### Management Commands

```bash
filesearch stats

filesearch verify <path>

filesearch watch <path>

filesearch clear --confirm

filesearch vacuum
```

#### Export

```bash
filesearch export --output results.json --query "*.rs"

filesearch export --output results.txt --query "test"
```

#### Interactive Mode

```bash
filesearch interactive
```

Interactive commands:

- `:help` - Show help
- `:stats` - Show index statistics
- `:history` - Show search history
- `:clear` - Clear screen
- `:quit` - Exit

### Configuration

Configuration can be loaded from TOML or JSON files:

```toml
[config]
index_path = "./filesearch.db"
thread_count = 8
max_file_size_for_content = 10485760  # 10MB
enable_content_search = true
enable_fuzzy_search = true
fuzzy_threshold = 0.7
cache_size = 1000
bloom_filter_capacity = 10000000
bloom_filter_error_rate = 0.0001
max_search_results = 1000
batch_size = 1000
follow_symlinks = false
index_hidden_files = false
exclusion_patterns = [".git", "node_modules", "target", ".DS_Store"]
watch_debounce_ms = 500
enable_access_tracking = true
db_pool_size = 10
```

Load configuration:

```rust
use rusty_files::SearchConfig;

let config = SearchConfig::from_file(&PathBuf::from("config.toml"))?;
let engine = SearchEngine::with_config("./index.db", config)?;
```

## Performance

### Benchmarks

Benchmark results from actual runs on the test environment:

#### Indexing Performance
| Files | Time (avg) | Throughput |
|-------|------------|------------|
| 100 files | 101.96 ms | ~981 files/sec |
| 500 files | 406.47 ms | ~1,230 files/sec |
| 1000 files | 1.0491 s | ~953 files/sec |
| Incremental update | 646.82 ms | N/A |

#### Search Performance
| Operation | Time (avg) | Description |
|-----------|------------|-------------|
| Simple search | 1.47 ms | Basic filename matching |
| Pattern search | 95.9 µs | Glob pattern matching |
| Fuzzy search | 1.25 ms | Fuzzy matching algorithm |
| Filtered search | 433.4 µs | Search with filters |
| Complex search | 1.22 ms | Multi-criteria search |

**Test Environment:** Linux 4.4.0, Release build with optimizations

**Notes:**
- Indexing performance scales well with parallel processing
- Search operations are sub-millisecond to low-millisecond range
- Pattern matching is extremely fast (~96 µs)
- Memory usage: <100MB base + configurable cache
- Startup time: <100ms with pre-built index

### Optimization Tips

1. **Adjust thread count**: Set `thread_count` to CPU cores × 2
2. **Tune cache size**: Larger cache = faster repeated searches
3. **Disable content search**: If you don't need it, disable for faster indexing
4. **Use exclusion patterns**: Skip unnecessary directories
5. **Batch operations**: Use batch indexing for large directories

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                     PUBLIC API LAYER                         │
│  ┌────────────┐  ┌──────────────┐  ┌──────────────────┐    │
│  │SearchEngine│  │ QueryBuilder │  │ ConfigManager    │    │
│  └────────────┘  └──────────────┘  └──────────────────┘    │
└────────────┬────────────────┬────────────────┬──────────────┘
             │                │                │
┌────────────▼────────────────▼────────────────▼──────────────┐
│                    CORE ENGINE LAYER                         │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐   │
│  │ Indexer  │  │ Searcher │  │ Watcher  │  │ Ranker   │   │
│  └──────────┘  └──────────┘  └──────────┘  └──────────┘   │
└────────────┬────────────┬────────────┬──────────────────────┘
             │            │            │
┌────────────▼────────────▼────────────▼──────────────────────┐
│                   DATA ACCESS LAYER                          │
│  ┌────────────┐  ┌──────────────┐  ┌──────────────────┐    │
│  │ Index DB   │  │ Cache Manager│  │ Bloom Filter     │    │
│  │ (SQLite)   │  │ (LRU)        │  │                  │    │
│  └────────────┘  └──────────────┘  └──────────────────┘    │
└──────────────────────────────────────────────────────────────┘
```

## Examples

See the `examples/` directory for complete examples:

- `basic_search.rs` - Simple file searching
- `async_indexing.rs` - Asynchronous indexing
- `custom_config.rs` - Custom configuration

Run examples:

```bash
cargo run --example basic_search
```

## Testing

Run tests:

```bash
cargo test

cargo test --all-features

cargo test --release
```

Run benchmarks:

```bash
cargo bench
```

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Acknowledgments

Built with:

- [tokio](https://tokio.rs/) - Async runtime
- [rusqlite](https://github.com/rusqlite/rusqlite) - SQLite bindings
- [walkdir](https://github.com/BurntSushi/walkdir) - Directory traversal
- [notify](https://github.com/notify-rs/notify) - File system watching
- [fuzzy-matcher](https://github.com/lotabout/fuzzy-matcher) - Fuzzy matching
- [clap](https://github.com/clap-rs/clap) - CLI parsing

## Support

For issues, questions, or suggestions, please [open an issue](https://github.com/Aryagorjipour/rusty-files/issues).