Module scanner 

Streaming scanner for detecting and replacing sensitive data.

Architecture

The streaming scanner processes input data in configurable chunks, detecting secret patterns (regex or literal) and applying one-way replacements via the MappingStore. This design supports files of 20–100 GB+ without requiring the entire content to fit in memory.

┌──────────────┐     ┌─────────────────┐     ┌──────────────────┐
│  Input (Read) │ ──▶ │  StreamScanner  │ ──▶ │  Output (Write)  │
│  (chunked)    │     │  (pattern match │     │  (sanitized)     │
└──────────────┘     │   + replace)    │     └──────────────────┘
                      └────────┬────────┘
                               │
                      ┌────────▼────────┐
                      │  MappingStore   │
                      │  (dedup cache)  │
                      └─────────────────┘

Chunk Overlap Strategy

To avoid missing matches that span chunk boundaries, the scanner maintains an overlap window between consecutive chunks:

1. Read chunk_size bytes of new data.
2. Prepend the carry buffer (tail of previous window).
3. Scan the combined window for all pattern matches.
4. Compute commit_point = window.len() - overlap_size, then adjust: a match fully inside the window that straddles the boundary moves the commit point up (so it is emitted whole); a match that runs to the right edge of the window may be truncated by the buffer, so the commit point is pulled back to its start and the whole match is carried.
5. Emit output for window[..commit_point] with replacements applied.
6. Set carry = window[commit_point..] for the next iteration.

Matches up to chunk_size bytes are therefore always seen in full before being committed. A single match longer than chunk_size (a pathological unbroken token) can never be buffered; rather than leak its tail, the scanner fails closed and replaces the whole run with a fixed redaction marker (see OVERLONG_MARKER).

Thread Safety

StreamScanner is Send + Sync. Multiple files can be scanned concurrently using a shared Arc<StreamScanner>, all backed by the same MappingStore for per-run dedup consistency.

Performance

• Chunk-based I/O: only chunk_size + overlap_size bytes in memory per active scan.
• Compiled regex: patterns are compiled once at construction and reused across all chunks and files.
• Lock-free reads: the DashMap inside MappingStore provides lock-free reads for already-seen values.
• File-level parallelism: share Arc<StreamScanner> across threads to scan multiple files concurrently.

Structs

MatchLocation

The file-level position of a single scanner match.

ScanConfig

Configuration for the streaming scanner.

ScanPattern

A pattern rule defining what to scan for and how to categorize matches.

ScanProgress

Progress snapshot emitted during streaming scans.

ScanStats

Statistics collected during a scan operation.

SecretsLoadResult

Result of loading a secrets file into a StreamScanner.

StreamScanner

Streaming scanner that detects and replaces sensitive patterns.

Constants

KV_LABEL_SUFFIX

Label suffix that marks patterns as key-value-only.