ZahirScan: Template-Based Content Compression & Metadata Extraction

"Others will dream that I am mad, while I dream of the Zahir." — JL Borges, Labyrinths

A high-performance Rust CLI that uses probabilistic template mining to extract structure and patterns from content, and metadata for the formats below.

Supported formats:

• Logs: Plain text logs, JSON-formatted logs, structured log files
• Text Documents: TXT, Markdown (MD), plain text content
• Documents: DOCX, XLSX, PPTX, PDF, EPUB
• Databases: SQLite (.db, .sqlite, .sqlite3)
• Settings: INI (.ini, .cfg), TOML (.toml, .lock), YAML (.yaml, .yml), XML (.xml)
• Structured text: CSV/TSV/tab/psv, JSON, HTML (.html, .htm)
• Tabular / columnar: Parquet; Arrow IPC / Feather (.arrow, .feather, .ipc); Avro; ORC; NumPy (.npy, .npz); HDF5 (.h5, .hdf5); NetCDF (.nc, .cdf); Matrix Market (.mtx); MATLAB (.mat); Zarr (.zarr)
• Models: ONNX (.onnx), GGUF (.gguf), TensorFlow Lite (.tflite), Safetensors (.safetensors)
• Archives: ZIP (.zip); TAR and compressed TAR (.tar, .tar.gz, .tgz, .tar.bz2, .tar.xz).
• Code/Scripts: e.g. .py, .rs, .js, .ts, .sh, Makefile, Dockerfile, etc
• Images: JPEG, PNG, GIF, WebP, BMP, TIFF
• Videos: MP4, MKV, AVI, MOV, WMV, FLV, WebM, M4V, 3GP, OGV
• Audio: MP3, FLAC, WAV, M4A, AAC, OGG, Opus, WMA, APE, DSD, DSF

Key Features

• Template mining: Repeated patterns in logs/text → templates with placeholders
• Memory-mapped I/O: memmap2; single open per path
• Adaptive parallelization: Phase 2 chunk sizes and Rayon batching follow Phase 1 stats; see Architecture and Configuration → adaptive batching
• Path batching: Large path lists run in fd-limit-sized batches so mmaps stay bounded; see Architecture
• Streamable output: OutputSink::Collect (default), OutputSink::StreamOnly (callback per file, no collection), or OutputSink::Channel (send on channel); compatible with batched scans
• Size reduction: Typically 80–95% smaller than raw while preserving structure and metadata

Metadata extraction by format

Installation

cargo add zahirscan        # library
cargo install zahirscan    # CLI

Optional system tools / libraries:

• ffprobe (FFmpeg): needed for rich video/audio metadata
• libnetcdf (NetCDF C): the netcdf crate links it at build time

Usage

CLI

$ zahirscan --help
Template mining for text/logs and metadata extraction for media, documents, archives, and more

Usage: zahirscan [OPTIONS] [COMMAND]

Commands:
  init  Write default config to XDG config dir (~/.config/zahirscan/zahirscan.toml or equivalent)
  help  Print this message or the help of the given subcommand(s)

Options:
  -i, --input <INPUT>...  Input file(s) to parse (can specify multiple)
  -o, --output <OUTPUT>   Output folder path (defaults to temp file if not specified). Creates filename.zahirscan.out in the folder for each input file
  -f, --full              Output mode: full metadata (for development/debugging). Default is templates-only mode (minimal JSON with templates & writing footprint)
  -d, --dev               Development mode: enables debug logging. Default is production mode (info level only). This disables progress bars if enabled
  -r, --redact            Redact file paths in output (show only filename as ***/filename.ext). Useful for privacy when sharing output JSON
  -n, --no-media          Skip media metadata extraction (audio, video, image). Faster processing when metadata is not needed
  -p, --progress          Show progress bars during processing. This is ignored if dev mode is enabled
  -h, --help              Print help
  -V, --version           Print version

Output formats:

• Mode 1 (Templates): Minimal JSON with templates, writing footprint (text/markdown), and per-format metadata where applicable—see Supported formats (media, code, logs, JSON, documents, tabular/columnar, archives, …)
• Mode 2 (Full): Mode 1 output plus:
  • File statistics (size, line count, processing time)
  • Size comparison (before/after)

Library Usage

ZahirScan can be used as a Rust library to extract schemas (templates and metadata) from files programmatically.

Basic: collect all outputs

use zahirscan::{extract_zahir, OutputMode, OutputSink};

// Default config, no file write; all results in result.outputs
let result = extract_zahir(
    "file.log",
    OutputMode::Full,
    None,
    None,
    &OutputSink::Collect,
)?;
// result.outputs, result.phase1_failed, result.phase2_failed

Stream-only: callback or channel (no collection, bounded memory)

use std::sync::{Arc, Mutex};
use zahirscan::{extract_zahir, Output, OutputMode, OutputSink};

let collected = Arc::new(Mutex::new(Vec::<(String, zahirscan::Output)>::new()));
let c = Arc::clone(&collected);
let sink = OutputSink::StreamOnly(Box::new(move |path, out| {
    c.lock().unwrap().push((path, out));
}));
let result = extract_zahir(
    ["a.log", "b.log"],
    OutputMode::Full,
    None,
    None,
    &sink,
)?;
// result.outputs is empty; results are in collected

Streaming input (paths from a channel)

use std::sync::mpsc;
use zahirscan::{extract_zahir_from_stream, OutputMode, OutputSink};

let (tx, rx) = mpsc::channel();
// Producer sends paths, then drops tx
let result = extract_zahir_from_stream(&rx, OutputMode::Full, None, None, &OutputSink::Collect)?;

Inputs: single path (&str, String) or multiple (&[&str], Vec<String>, etc.). Config: pass Some(&config) for custom RuntimeConfig; None uses embedded default. Full API: ZahirScanResult, Output, OutputSink, Template, WritingFootprint, and per-format metadata: docs.rs.

Configuration

• CLI: Embedded default (config.toml), merged with a user config in the app data dir if present. Only keys in the user file override.
  • User config path: Unix ~/.config/zahirscan/zahirscan.toml (or $XDG_CONFIG_HOME/zahirscan/zahirscan.toml), Windows %APPDATA%\zahirscan\zahirscan.toml.
  • Run zahirscan init to write the embedded default to edit; the CLI will use it as the overlay.
• Library: extract_zahir(..., config: None, output_dir: None) uses the embedded default only (no overlay). For custom config pass Some(&config); RuntimeConfig::new() is the embedded default (no file I/O). Overlay is only used by the CLI via setup::load_config().

Full schema: config.toml.

Adaptive batching and parallelization (see Architecture for how Phase 1 / Phase 2 interact with batches):

• Phase 2 adaptive chunking: Chunk sizes and “chunks per file” come from Phase 1 stats (file count, mean bytes, variance); targets a neat multiple of max_workers for load balancing.
• Phase 2 parallel batching: When task count exceeds workers × threshold_multiplier, Rayon uses with_min_len(batch_size) to avoid thread-pool saturation; otherwise full parallelism.
• max_workers = 0 uses a sensible default (e.g. num_cpus - 1). No manual tuning is required for typical workloads.

File filtering ([filter]):

• ignore_patterns: skip files whose basename matches (exact: .DS_Store, Thumbs.db; suffix: *.swp, *~; prefix: prefix*)
• ignore_hidden_files = true: skip Unix hidden files (basename starts with .)

Architecture

Phase 1: Format detection, stats (lines/bytes/tokens), mmap per path, content-type classification. Runs in parallel over paths (Rayon).

Path batching: When there are more paths than the batch size (from the process fd limit), each chunk runs Phase 1 then Phase 2; the chunk and its mmaps are dropped before the next chunk.

Phase 2: Metadata extraction per format, template mining, writing footprint (exact-pattern then shape fallback for text/markdown). One Rayon pool; chunk sizing and with_min_len batching follow Phase 1 stats—tunable fields are under Configuration → adaptive batching.

Security

Read-only, non-invasive: path sanitization, existence checks, no source modification.

License

Dual-licensed under MIT or Apache-2.0.