scankit

Walk + watch + filter directory trees. The shared scanner Tauri / Iced / native desktop apps reach for when they need to enumerate user files.

Status: v0.1 — one-shot directory walking with exclude-glob and size-cap filters, backed by walkdir + globset. Continuous watch (filesystem-event monitoring on top of an initial walk) behind the watch feature lands in v0.2 — the feature flag exists in v0.1 as a no-op so consumers can pin against the eventual shape.

Why this exists

Every "index files on the user's machine" project — RAG tools, search apps, backup utilities, file watchers, document assistants — rebuilds the same five hundred lines of walkdir-with-excludes-and- size-cap-and-symlink-handling glue. Every project gets it slightly wrong:

• Missed **/.git/** in the exclude set, scanned 200K objects in a .pack file.
• Forgot to cap file sizes, OOM'd on a 50 GB sqlite database the user accidentally dropped in their Documents folder.
• Followed a symlink loop and hung the indexer.
• Rebuilt the GlobSet per-iteration, ate 30 % of CPU on glob compilation alone.

scankit ships these bits once, with the edge cases handled in one place. It's deliberately lower-level than a full indexer — it does not parse files, generate embeddings, or persist anything. It hands you ScanEntrys and gets out of the way. Pair it with mdkit for documents → markdown, with calamine / csv for tabular files, with whatever you like for the rest.

Quick start

use scankit::{Scanner, ScanConfig};
use std::path::Path;

let scanner = Scanner::new(
    ScanConfig::default()
        .max_file_size_bytes(50 * 1024 * 1024) // 50 MB cap
        .add_exclude("**/.git/**")?
        .add_exclude("**/node_modules/**")?
        .add_exclude("**/.DS_Store")?,
)?;

for result in scanner.walk(Path::new("/Users/me/Documents")) {
    match result {
        Ok(entry) => println!(
            "{}: {} bytes, .{}",
            entry.path.display(),
            entry.size_bytes,
            entry.extension,
        ),
        Err(e) => eprintln!("scan error: {e}"),
    }
}
# Ok::<(), scankit::Error>(())

Design principles

1. Do one thing well. Walk + filter + emit ScanEntry. Anything richer (parse, embed, persist) is the consuming application's job.
2. Send + Sync everywhere. A single Scanner shared across threads, a single GlobSet built once.
3. No surprises in the iterator. Filtered-out entries are silently dropped. Errors come through as Err items in the stream — callers can log-and-continue or short-circuit.
4. Forward-compat defaults. ScanConfig and ScanEntry are #[non_exhaustive] so we can add fields (content hash, inode, per-entry metadata) without breaking downstream callers.
5. Honest dep budget. walkdir + globset + thiserror are the only required deps. notify is gated behind the watch feature.

Feature flags

License

Dual-licensed under MIT OR Apache 2.0 at your option. SPDX: MIT OR Apache-2.0.

Status & roadmap

• v0.1 — one-shot walking. Scanner + ScanConfig + ScanEntry, exclude-glob and size-cap filters, symlink handling, lazy iterator. The watch feature exists as a no-op placeholder so the shape is stable.
• v0.2 — watch feature: continuous filesystem-event monitoring on top of an initial walk, backed by notify.
• v0.3 — extension-based dispatch helper (provide a HashMap<&str, Box<dyn Handler>>, get back a stream of handler-processed outputs).
• v0.4 — audit pass + first stable trait release (1.0 candidate).

Issues, PRs, and design discussion welcome at https://github.com/seryai/scankit/issues.

Used by

scankit was extracted from the folder-scanner of Sery Link, a privacy-respecting data network for the files on your machines. If you use scankit in your project, please open a PR to add yourself here.

Acknowledgements

• walkdir — BurntSushi's battle-tested directory walker. Loop detection, permission handling, and Send-iterator semantics all come from there.
• globset — also BurntSushi's. The compiled multi-pattern glob matcher that makes our exclude set efficient even with hundreds of patterns.
• notify — the cross-platform filesystem-event crate that v0.2's watch loop will be built on.
• mdkit — sibling crate; scankit does "files → entries", mdkit does "documents → markdown".