Expand description
seen.db: the dedup store described in DESIGN.md §“Implementation”.
Keyed by (kind, dedup_key) (e.g. (event, <iCalendar UID>) or
(parcel, <trackingNumber>)), value is the content hash of the
last-filed payload. The pipeline checks the store before re-issuing
the expensive upstream calls (CalDAV PUT, Karrio register, 17track
register, Firefly POST/PUT) so a replay against a backlog doesn’t
pummel third-party APIs for artifacts we’ve already filed.
Local-filesystem targets (bills / receipts / tickets / subscriptions / local events) are intentionally not gated: rewriting a small JSON file is cheap and already idempotent (same key → same path), and not consulting the store means a corrupt store can’t make us miss an update.
§Storage
redb: pure-Rust, single-file, ACID. The whole API surface we
use is open, begin_read / begin_write, and one table lookup
per call. Concurrent milter tasks coordinate via redb’s
single-writer / many-reader model; a write transaction is
microsecond-scale so contention is a non-issue at our message rate.
Worst case if the store is corrupt or missing: we re-issue an upstream call. Each gated target is idempotent (CalDAV PUT replaces; Karrio/17track register are duplicate-safe; Firefly does update-or-create), so a stale store costs at most some wasted network round-trips, never lost data.
Structs§
- Store
- The dedup store. Cloneable: wraps an
Arc<Database>so the milter’s per-message tasks share a single open file handle.
Enums§
- Kind
- Artifact kinds the dedup store distinguishes. Used as the composite-key prefix; new kinds get a new variant rather than a free-form string so we can grep for callers.
Functions§
- hash
- Compute the content hash we store. Lowercase hex SHA-256; cheap to generate, stable across mailsift restarts, and small enough that the whole store stays well under a megabyte even with thousands of entries.