Lynceus

Lynceus is a lightweight, high-performance directory file watcher written in Rust. It is specifically designed to reliably monitor directories on network shares (SMB/CIFS mounts) where native OS filesystem events (like inotify or FSEvents) are either unavailable or fail to report events correctly.

Lynceus tracks newly created files and ensures that writing/copying processes have completely finished before reporting the file as created.

Features

• Network-Share Optimized: Uses poll-based directory watching with content hashing/comparison to guarantee event capture across network mounts.
• Concurrent Cooldown Checks: Multiplexes up to 100 concurrent stability checks using an extremely lightweight Tokio async stream architecture. It automatically waits for large files to finish copying without blocking other detections.
• Glob Pattern Filtering: Filter detected files by standard glob patterns (e.g. **/*.mp4).
• Flexible Webhook Customization: Define custom JSON payloads utilizing the rich Liquid template engine syntax (supporting filters like upcase, split, and last).
• Resilient Delivery: Dispatches webhook notifications asynchronously and automatically retries transient failures with exponential backoff.
• Fully Nix-Integrated: Minimal OCI container images, formatting via nix fmt (using treefmt), and comprehensive CI checking.

Three-Stage Processing Pipeline

Lynceus processes file system events sequentially through three highly optimized stages:

graph TD
    %% Watcher Section
    subgraph Watcher ["1. Directory Watcher & Filter"]
        A["Target Directory (SMB/CIFS)"] -->|Interval-based Polling| B["Detect Changed/New Files"]
        B --> C{"Glob Pattern Filter?"}
        C -->|Does Not Match| D["Ignore Event"]
        C -->|Matches| E["Forward to Stream"]
    end

    %% Stabilizer Section
    subgraph Stabilizer ["2. File Stabilizer (Concurrent)"]
        E --> F["Wait Queue (.buffer_unordered)"]
        F --> G["Initiate Cooldown Timer"]
        G --> H["Check Size & Mod Time"]
        H --> I{"Stable over Cooldown?"}
        I -->|No / Modifying| G
        I -->|Error limit exceeded| K["Discard (Timeout)"]
        I -->|Yes, stable-count reached| L["Emit 'File Created' Event"]
    end

    %% Webhook Section
    subgraph Webhook ["3. Webhook Dispatcher (Non-Blocking)"]
        L --> M["Format Payload (Liquid Templates)"]
        M --> N["Spawn Async Task (TaskTracker)"]
        N --> O["POST HTTP request"]
        O --> P{"Response Success?"}
        P -->|No & Retries Available| Q["Exponential Backoff"] --> N
        P -->|No & Max Retries Exceeded| R["Log Error & Drop"]
        P -->|Yes| S["Success (Log Status)"]
    end

    %% Styling Elements for Visual Appeal
    classDef watcherStyle fill:#f0f8ff,stroke:#0066cc,stroke-width:1px,color:#003366;
    classDef stabilizerStyle fill:#fffcf0,stroke:#cc9900,stroke-width:1px,color:#664400;
    classDef webhookStyle fill:#f0fff0,stroke:#00cc66,stroke-width:1px,color:#004400;
    classDef processNode fill:#ffffff,stroke:#333333,stroke-width:1px;

    class A,B,C,D,E processNode;
    class F,G,H,I,K,L processNode;
    class M,N,O,P,Q,R,S processNode;

    class Watcher watcherStyle;
    class Stabilizer stabilizerStyle;
    class Webhook webhookStyle;

1. Directory Watcher & Filter

• Interval Polling: Regularly scans the target watch directory to detect new files, handling network shares where native event APIs (e.g. inotify, FSEvents) fall short.
• Debouncing: Debounces incoming events to prevent overwhelming the processing queue.
• Pattern Matching: Applies glob filters (e.g., --pattern "**/*.txt") early on to immediately discard irrelevant files.

2. File Stabilizer

• Concurrent Monitoring: Uses an async queue (.buffer_unordered(100)) to concurrently track up to 100 files simultaneously without blocking other detections.
• Stability Heuristic: Measures file size and modification times across successive cooldown intervals. A file is declared fully written only after surviving stable-count consecutive checks without size or modification changes.
• Resilience: If a file check fails continuously (e.g., locked file, network drop), it automatically times out and drops the event after error-count consecutive errors.

3. Webhook Dispatcher

• Liquid Rendering: Generates flexible JSON payloads dynamically using the Liquid template engine based on the --webhook-template option.
• Non-blocking Dispatch: Dispatches events asynchronously via background tasks (tokio::spawn), isolated from the main watching loop.
• Exponential Backoff: Resilient HTTP POST delivery that retries transient failures with progressive backoff.
• Graceful Shutdown: Integrates with a TaskTracker to guarantee pending webhooks are fully drained upon termination.

Installation

Ensure you have Rust installed (MSRV 1.85+), then clone the repository and build the binary:

cargo build --release

Usage

Run the compiled binary by passing the target directory path as a positional argument:

./target/release/lynceus /path/to/watch

Configuration Options

Usage: lynceus [OPTIONS] <PATH> [WEBHOOK_URL]

Arguments:
  <PATH>         Path to watch for changes [env: LYNCEUS_PATH=]
  [WEBHOOK_URL]  Optional webhook URL to post a message to when a file is created [env: LYNCEUS_WEBHOOK_URL=]

Options:
  -p, --pattern <PATTERN>
          Optional glob pattern relative to the watch path to filter created files (e.g. "**/*.txt") [env: LYNCEUS_PATTERN=]
      --webhook-template <WEBHOOK_TEMPLATE>
          Optional JSON template for the webhook payload. Supports `{{path}}`, `{{type}}`, and `{{timestamp}}` placeholders [env: LYNCEUS_WEBHOOK_TEMPLATE=] [default: {"type":"{{type}}","timestamp":"{{timestamp}}","path":"{{path}}"}]
      --webhook-retries <WEBHOOK_RETRIES>
          Number of retries when sending a webhook fails [env: LYNCEUS_WEBHOOK_RETRIES=] [default: 3]
  -i, --interval <INTERVAL>
          Polling interval (e.g. 2s, 500ms) [env: LYNCEUS_INTERVAL=] [default: 2s]
  -d, --debounce <DEBOUNCE>
          Debounce duration (e.g. 5s, 10s) [env: LYNCEUS_DEBOUNCE=] [default: 5s]
  -c, --cooldown <COOLDOWN>
          Cooldown interval for checking file stability (e.g. 10s, 30s) [env: LYNCEUS_COOLDOWN=] [default: 10s]
  -s, --stable-count <STABLE_COUNT>
          Number of consecutive stable checks required to consider the file created [env: LYNCEUS_STABLE_COUNT=] [default: 3]
  -e, --error-count <ERROR_COUNT>
          Number of consecutive error checks before timing out/giving up on the file [env: LYNCEUS_ERROR_COUNT=] [default: 5]
  -h, --help
          Print help
  -V, --version
          Print version

Examples & Common Configurations

1. Glob Pattern Filtering

If you only care about specific files (e.g., media files), you can filter incoming events using glob syntax:

cargo run --release -- /path/to/watch --pattern "**/*.{mp4,mkv}"

2. Robust Network Copy Detection

For network copies (e.g. copying huge files over a slow SMB share), we want a longer stability cooldown. You can run Lynceus to poll every 5 seconds, debounce events for 15 seconds, and check file stability every 10 seconds:

cargo run --release -- /path/to/watch --interval 5s --debounce 15s --cooldown 10s

3. Customizable Webhook Notifications

You can specify an optional Discord/Slack or generic HTTP endpoint webhook. Webhooks are dispatched in the background and do not block the primary watcher loop.

Liquid-syntax Template Engine

Using the --webhook-template flag (or LYNCEUS_WEBHOOK_TEMPLATE env var), you can customize the JSON payload. Lynceus supports standard Liquid tags and filters.

• Placeholders:
  • {{path}}: Relative path of the created file.
  • {{type}}: Event type (e.g. "file.created").
  • {{timestamp}}: Event timestamp in RFC 3339 format (e.g. "2026-05-28T15:02:50Z").
• Liquid Filters: Extract filenames or transform text using filters (e.g., {{path | split: '/' | last}} extracts only the filename).

Example (Slack-compatible webhook message):

cargo run --release -- /path/to/watch https://hooks.slack.com/services/... \
  --webhook-template '{"text": "New file created: {{path | split: '\''/'\'' | last}} at {{path}}"}'

Transient Failure Retries

Transient network or server errors are automatically retried using an exponential backoff policy (defaults to 3 retries) before declaring failure.

Logging & Diagnostics

Lynceus supports the standard RUST_LOG environment variable to configure logging levels.

Standard Info logs (Default)

$ cargo run -- /path/to/watch
2026-05-27T08:00:00Z  INFO lynceus: Starting Lynceus args=Args { path: "/path/to/watch", pattern: None, webhook_url: None, webhook_template: Object {"path": String("{{path}}"), "timestamp": String("{{timestamp}}"), "type": String("{{type}}")}, webhook_retries: 3, interval: Duration(2s), debounce: Duration(5s), cooldown: Duration(10s), stable_count: 3, error_count: 5 }
2026-05-27T08:00:00Z  INFO lynceus: Watching for new files watch_path="/path/to/watch"
2026-05-27T08:00:35Z  INFO lynceus: File created path="video.mp4"

Detailed Debug logs

To see real-time stability polling ticks:

$ RUST_LOG=debug cargo run -- /path/to/watch
2026-05-27T08:00:05Z DEBUG lynceus: New file detected, waiting for write to complete path="video.mp4"
2026-05-27T08:00:15Z DEBUG lynceus: File is stable path="video.mp4" size="10.00 MiB" stable_count=1
2026-05-27T08:00:25Z DEBUG lynceus: File is stable path="video.mp4" size="10.00 MiB" stable_count=2
2026-05-27T08:00:35Z DEBUG lynceus: File is stable path="video.mp4" size="10.00 MiB" stable_count=3
2026-05-27T08:00:35Z  INFO lynceus: File created path="video.mp4"

CI/CD & Development

Lynceus leverages Nix for fully reproducible formatting, builds, and sandbox checks:

• Format Code: Formats Rust (via rustfmt) and Nix code globally.

  nix fmt
  

• Flake Checks: Validates formatting (treefmt), cargo check, clippy analysis (--deny warnings), and the cargo test suite in a hermetic environment.

  nix flake check
  

• Minimal OCI Container: Builds a minimal, highly secure OCI/Docker container.

  nix build .#image