Skip to main content

Module store

sanitize_engine

Module store 

Source
Expand description

Thread-safe, concurrent one-way replacement store.

§Concurrency Model

The store uses dashmap::DashMap — a concurrent hash map with shard-level locking (default 64 shards). This gives us:

  • Lock-free reads for lookups of already-mapped values.
  • Shard-level write locks that are held only while inserting a new entry. With 64 shards and 8–16 threads, the probability of two threads contending on the same shard is very low.
  • Atomic get-or-insert via the entry() API, which prevents TOCTOU races and guarantees first-writer-wins semantics.

§Structure

The forward map is two-level: Category → original → sanitized.

DashMap<Category, Arc<DashMap<ZeroizingString, (CompactString, usize)>>>
   outer (~20 entries, always hot in cache)
              └── inner (one per category, holds the actual values)

This lets the fast-path read call inner.get(original: &str) without constructing a temporary String, because ZeroizingString: Borrow<str>. For files where the same value appears thousands of times, this eliminates thousands of malloc/free cycles on the hot path.

Replacements are one-way only — there is no reverse map, no mapping file, and no restore capability.

§Memory Characteristics

At 10M unique values with average key length 20 bytes and average value length 30 bytes:

  • Forward map: 10M × (20 + 30 + ~120 DashMap overhead) ≈ 1.7 GB
  • Total: ~1.7 GB — acceptable for server workloads.

An optional capacity_limit can be set to prevent unbounded growth.

Structs§

MappingStore
Thread-safe concurrent one-way replacement store.