lynceus 0.4.2

lynceus is a file watcher that reports file changes using a webhook.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149

# Lynceus

Lynceus is a lightweight, high-performance directory file watcher written in Rust. It is specifically designed to reliably monitor directories on **network shares (SMB/CIFS mounts)** where native OS filesystem events (like `inotify` or `FSEvents`) are either unavailable or fail to report events correctly.

Lynceus tracks newly created files and ensures that writing/copying processes have **completely finished** before reporting the file as created.

## Features

- **Network-Share Optimized**: Uses poll-based directory watching with content hashing/comparison to guarantee event capture across network mounts.
- **Concurrent Cooldown Checks**: Multiplexes up to 100 concurrent stability checks using an extremely lightweight Tokio async stream architecture. It automatically waits for large files to finish copying without blocking other detections.
- **Glob Pattern Filtering**: Filter detected files by standard glob patterns (e.g. `**/*.mp4`).
- **Flexible Webhook Customization**: Define custom JSON payloads utilizing the rich **Liquid template engine** syntax (supporting filters like `upcase`, `split`, and `last`).
- **Resilient Delivery**: Dispatches webhook notifications asynchronously and automatically retries transient failures with **exponential backoff**.
- **Fully Nix-Integrated**: Minimal OCI container images, formatting via `nix fmt` (using `treefmt`), and comprehensive CI checking.

---

## Installation

Ensure you have Rust installed (MSRV 1.85+), then clone the repository and build the binary:

```bash
cargo build --release
```

---

## Usage

Run the compiled binary by passing the target directory path as a positional argument:

```bash
./target/release/lynceus /path/to/watch
```

### Configuration Options

```text
Usage: lynceus [OPTIONS] <PATH> [WEBHOOK_URL]

Arguments:
  <PATH>         Path to watch for changes [env: LYNCEUS_PATH=]
  [WEBHOOK_URL]  Optional webhook URL to post a message to when a file is created [env: LYNCEUS_WEBHOOK_URL=]

Options:
  -p, --pattern <PATTERN>
          Optional glob pattern relative to the watch path to filter created files (e.g. "**/*.txt") [env: LYNCEUS_PATTERN=]
      --webhook-template <WEBHOOK_TEMPLATE>
          Optional JSON template for the webhook payload. Supports `{{path}}`, `{{type}}`, and `{{timestamp}}` placeholders [env: LYNCEUS_WEBHOOK_TEMPLATE=] [default: {"type":"{{type}}","timestamp":"{{timestamp}}","path":"{{path}}"}]
      --webhook-retries <WEBHOOK_RETRIES>
          Number of retries when sending a webhook fails [env: LYNCEUS_WEBHOOK_RETRIES=] [default: 3]
  -i, --interval <INTERVAL>
          Polling interval (e.g. 2s, 500ms) [env: LYNCEUS_INTERVAL=] [default: 2s]
  -d, --debounce <DEBOUNCE>
          Debounce duration (e.g. 5s, 10s) [env: LYNCEUS_DEBOUNCE=] [default: 5s]
  -c, --cooldown <COOLDOWN>
          Cooldown interval for checking file stability (e.g. 10s, 30s) [env: LYNCEUS_COOLDOWN=] [default: 10s]
  -s, --stable-count <STABLE_COUNT>
          Number of consecutive stable checks required to consider the file created [env: LYNCEUS_STABLE_COUNT=] [default: 3]
  -e, --error-count <ERROR_COUNT>
          Number of consecutive error checks before timing out/giving up on the file [env: LYNCEUS_ERROR_COUNT=] [default: 5]
  -h, --help
          Print help
  -V, --version
          Print version
```

---

## Examples & Common Configurations

### 1. Glob Pattern Filtering
If you only care about specific files (e.g., media files), you can filter incoming events using glob syntax:

```bash
cargo run --release -- /path/to/watch --pattern "**/*.{mp4,mkv}"
```

### 2. Robust Network Copy Detection
For network copies (e.g. copying huge files over a slow SMB share), we want a longer stability cooldown. You can run Lynceus to poll every 5 seconds, debounce events for 15 seconds, and check file stability every 10 seconds:

```bash
cargo run --release -- /path/to/watch --interval 5s --debounce 15s --cooldown 10s
```

### 3. Customizable Webhook Notifications
You can specify an optional Discord/Slack or generic HTTP endpoint webhook. Webhooks are dispatched in the background and do not block the primary watcher loop.

#### Liquid-syntax Template Engine
Using the `--webhook-template` flag (or `LYNCEUS_WEBHOOK_TEMPLATE` env var), you can customize the JSON payload. Lynceus supports standard Liquid tags and filters.

* **Placeholders**:
  * `{{path}}`: Relative path of the created file.
  * `{{type}}`: Event type (e.g. `"file.created"`).
  * `{{timestamp}}`: Event timestamp in RFC 3339 format (e.g. `"2026-05-28T15:02:50Z"`).
* **Liquid Filters**: Extract filenames or transform text using filters (e.g., `{{path | split: '/' | last}}` extracts only the filename).

**Example (Slack-compatible webhook message)**:
```bash
cargo run --release -- /path/to/watch https://hooks.slack.com/services/... \
  --webhook-template '{"text": "New file created: {{path | split: '\''/'\'' | last}} at {{path}}"}'
```

#### Transient Failure Retries
Transient network or server errors are automatically retried using an exponential backoff policy (defaults to 3 retries) before declaring failure.

---

## Logging & Diagnostics

Lynceus supports the standard `RUST_LOG` environment variable to configure logging levels. 

### Standard Info logs (Default)
```bash
$ cargo run -- /path/to/watch
2026-05-27T08:00:00Z  INFO lynceus: Starting Lynceus args=Args { path: "/path/to/watch", pattern: None, webhook_url: None, webhook_template: Object {"path": String("{{path}}"), "timestamp": String("{{timestamp}}"), "type": String("{{type}}")}, webhook_retries: 3, interval: Duration(2s), debounce: Duration(5s), cooldown: Duration(10s), stable_count: 3, error_count: 5 }
2026-05-27T08:00:00Z  INFO lynceus: Watching for new files watch_path="/path/to/watch"
2026-05-27T08:00:35Z  INFO lynceus: File created path="video.mp4"
```

### Detailed Debug logs
To see real-time stability polling ticks:
```bash
$ RUST_LOG=debug cargo run -- /path/to/watch
2026-05-27T08:00:05Z DEBUG lynceus: New file detected, waiting for write to complete path="video.mp4"
2026-05-27T08:00:15Z DEBUG lynceus: File is stable path="video.mp4" size="10.00 MiB" stable_count=1
2026-05-27T08:00:25Z DEBUG lynceus: File is stable path="video.mp4" size="10.00 MiB" stable_count=2
2026-05-27T08:00:35Z DEBUG lynceus: File is stable path="video.mp4" size="10.00 MiB" stable_count=3
2026-05-27T08:00:35Z  INFO lynceus: File created path="video.mp4"
```

---

## CI/CD & Development

Lynceus leverages **Nix** for fully reproducible formatting, builds, and sandbox checks:

- **Format Code**: Formats Rust (via `rustfmt`) and Nix code globally.
  ```bash
  nix fmt
  ```
- **Flake Checks**: Validates formatting (treefmt), cargo check, clippy analysis (`--deny warnings`), and the cargo test suite in a hermetic environment.
  ```bash
  nix flake check
  ```
- **Minimal OCI Container**: Builds a minimal, highly secure OCI/Docker container.
  ```bash
  nix build .#image
  ```