Module dedup 

In-process content-hash dedup for memory-store hooks and compactors.

Rig’s DemotionHook / Compactor traits require implementations to be idempotent on (conversation_id, messages). Append-only backends (such as .mv2 archives) have no unique-key enforcement, so this module provides a small content-hash gate that lives alongside the hook/compactor instance and prevents the same entry from being appended twice within a single process lifetime.

Scope of the guarantee

• Within a process: invoking the same operation more than once with the same (kind, conversation_id, role, scope, text) produces exactly one entry in the backing store.
• Across process restarts: dedup state is not persisted by default. Callers that need cross-restart idempotency can snapshot the set via DedupSet::snapshot before shutdown and replay it into a fresh instance via DedupSet::extend_from_snapshot.

Example

use rig_memory_policy::dedup::{DedupSet, compute_key};

let set = DedupSet::new();
let key = compute_key("demoted_message", "conv-1", "user", None, "hello");
assert!(!set.contains(&key).unwrap());
set.insert(key).unwrap();
assert!(set.contains(&key).unwrap());

Structs

DedupSet

In-memory set of dedup keys with snapshot / restore for opt-in cross-process persistence.

Functions

compute_key

Compute the dedup key for a single entry.

hex_encode_key

Hex-encode a DedupKey as 64 lowercase ASCII chars. Callers stamp this into per-entry metadata so the dedup decision survives in the archive.

Type Aliases

DedupKey

32-byte content fingerprint produced by blake3::hash.