Crate keyhog_scanner 

KeyHog Scanner: A high-performance, multi-layered secret detection engine.

This crate implements the core scanning logic, combining SIMD pre-filtering, Aho-Corasick literal matching, regex fallback, and ML-based confidence scoring.

Module map (by pipeline stage)

The modules below are declared in dependency order, but they READ in pipeline order, the same bytes→finding flow as [docs/src/architecture.md] and the method-level map in [engine] (engine::mod “# The one flow”). To find a responsibility, locate its stage:

• Config / state / shared types: [scanner_config], [scan_state], [types], hw_probe (hardware routing), [error].
• Phase 1 · prefilter (cheap “could a detector fire here?”) [alphabet_filter], [bigram_bloom], [prefix_trie], ascii_ci, simd / simdsieve_prefilter (feature-gated), prefilter_degrade (loud Law-10 fallback).
• Compile and lifecycle (detectors → matchers): compiled_scanner/, [compiler], shared_regexes, [static_intern].
• Scan engine (phase 1 triggers + phase 2 extraction; CPU or GPU): [engine] (start at its header doc), [pipeline], gpu. Public scan entry methods live in compiled_scanner/runtime.rs and dispatch here.
• Decode-through (nested base64/hex/url/unicode, recursive) decode, [decode_structure].
• Entropy: entropy is now the single home for all of it: the keyword/scanner detection logic plus the fast Shannon-entropy primitive entropy::fast (+ entropy::avx512 / entropy::fast_x86 / entropy::fast_neon SIMD impls, arch-gated).
• Confidence / ML: ml_scorer (serves the embedded weights.bin; trained out-of-band by the repo’s ml/), [confidence], probabilistic_gate.
• Context, fragment reassembly, multiline, suppression, resolution context, fragment_cache, [multiline], suppression, resolution, structured.
• Specialized validators: checksum, jwt, aws, homoglyph, [unicode_hardening].
• Cross-cutting: platform_compat, placeholder_words, telemetry, util_hash.

Most single-file modules are one responsibility each; the multi-file engine is the exception and carries its own internal map in engine::mod.

Re-exports

pub use crate::hw_probe::probe_hardware;

pub use crate::hw_probe::select_backend;

pub use crate::hw_probe::HardwareCaps;

pub use crate::hw_probe::ScanBackend;

pub use crate::util_hash::FNV_OFFSET_BASIS;

pub use crate::util_hash::FNV_PRIME;

Modules

aws

Offline AWS account-ID recovery from an access-key ID (no network/verify). Offline AWS account-ID recovery + canary-token classification.

checksum

Service-specific credential checksum validation (GitHub, npm, Slack, etc.). Checksum-aware credential validation.

context

Code context analysis (comments, assignments, test files). Structural context analysis: understand WHERE in code a potential secret appears.

decode

Decode-through pipeline for nested encodings (base64, hex, URL, etc.). Decode-through scanning: decode encoded strings before pattern matching.

entropy

Shannon entropy analysis for secret detection. Shannon entropy analysis for distinguishing secrets from ordinary text.

gpu

GPU-accelerated matching via wgpu. GPU-accelerated batch inference for the MoE classifier via wgpu compute shaders.

hw_probe

Hardware capability detection and backend selection. Hardware capability probing with once-cached results.

jwt

JWT structural validation and anomaly detection. JWT structural validation.

ml_scorer

Machine learning inference for secret scoring. ML-based secret scoring with a tiny mixture-of-experts network.

resolution

Match resolution and deduplication. Match resolution: when multiple detectors match the same region, keep only the most specific, highest-confidence match. Eliminates duplicates.

telemetry

Per-scan telemetry: always-on counters + opt-in --dogfood events. Lightweight per-scan telemetry.

util_hash

Shared FNV-1a hash + content-keyed memoization primitives. Single home for the seed every per-scan cache keys on, plus the bounded thread-local cache helper they all share, so a hash change can never re-key only some caches. Shared non-cryptographic hash + content-keyed memoization primitives.

Structs

BackendRecoveryReceipt

Complete, non-secret receipt for automatic recovery of stable input bytes.

BigramPrefilterCorpusStatus

Rejection effectiveness measured over one explicitly named input corpus.

BigramPrefilterStatus

Build-time hash-table density diagnostics for the Layer-0.5 prefilter.

CoalescedScanOutcome

Result of one fallible coalesced dispatch, including any completed recovery.

CompiledScanner

CompiledScannerRuntime

GpuBackendAvailability

GpuBackendCandidateStatus

GpuLiteralArtifact

Serialized VYRE literal matcher plus the cache identity used by runtime.

GpuLiteralArtifacts

The runtime GPU presence matcher artifacts derivable without a GPU device.

Phase1AdmissionPlan

Exact per-chunk phase-1 admissions computed while an autoroute key is built. The plan is intentionally opaque: callers can only reuse it through the scanner boundary that verifies its internal totals and exact live chunk identity. GPU region presence does not consume this plan because VYRE owns that path’s trigger admission.

Phase1AdmissionSummary

Exact direct-literal admission totals for one routed scan batch.

Phase2KeywordTriggerSummary

Exact phase-2 keyword-trigger density for one routed scan batch.

RecoveredInputRange

One exact source-byte interval completed after the selected backend faulted.

ScanExecutionRoute

Recall-equivalent execution choices resolved for one scan request.

ScannerConfig

Scanner-side configuration: the canonical ScanConfig, the single owned source of truth for every shared detection knob (decode depth, entropy, ML, confidence floor, keyword lists, …). PLUS the two knobs that are scanner-crate-local and have no place on keyhog-core’s ScanConfig:

ScannerTuningConfig

Explicit per-scanner performance-route tuning.

Enums

BigramPrefilterState

Operator-visible health state for the Layer-0.5 selective anchor prefilter.

GpuInitPolicy

ScanError

Errors returned while compiling detector patterns into a scanner.

Functions

compile_gpu_literal_artifacts

Compatibility entry point for callers that supplied Hyperscan tuning. GPU literal artifacts now depend only on the canonical detector plan.

compile_gpu_literal_artifacts_default

Compile GPU literal artifacts from the canonical detector plan.

gpu_batch_input_limit

VRAM-adaptive GPU batch-input limit. Bigger buffers mean fewer device dispatches per multi-TB scan; each kernel launch is a fixed ~50-300 µs cost regardless of payload, so doubling the input halves dispatch overhead. Capped by host VRAM (input + transition tables + match output must fit) and by a 1 GiB upper bound so the pre-compile time stays bounded.

gpu_batch_input_limit_bounds

The [floor, cap] the resolved GPU batch input limit is clamped into: the 128 MiB unknown-host floor and the 1 GiB pre-compile-time ceiling that bound the VRAM table. A Tier-A override is clamped into this range so no config/CLI value can request a buffer the sizing contract forbids.

gpu_literal_artifact_cache_dir

Canonical runtime directory for serialized GPU literal matcher artifacts.

is_entropy_detector

True when detector_id names the pure-entropy fallback family ("entropy" or any "entropy-*" id such as entropy-token).

is_generic_or_entropy_detector

True for a detector that fires via the entropy / phase2-generic path (the generic-* family + entropy fallback), carrying ZERO patterns by design.

profile_dump

Print and reset the unified profile tree. Safe to call when profiling was off (prints a single “disabled” line).

profile_reset

Discard all accumulated counters without printing (warm-up between runs).

regex_dfa_limit_default

The compiled-default per-regex DFA size limit ([REGEX_SIZE_LIMIT_BYTES]): the cap that takes effect when no --regex-dfa-limit / regex_dfa_limit override is set. Exposed so keyhog config --effective can report the real active default instead of a misleading “off” - an unset limit is never truly off, it falls back to this compiled cap.

set_gpu_batch_input_limit

Override the GPU batch input limit for this process. Call before scanning. 0 resets to the VRAM-adaptive default; any other value is clamped into gpu_batch_input_limit_bounds at read time. Tier-A config knob (compiled default -> TOML -> CLI), the sizing analogue of crate::types::set_regex_dfa_limit.

set_perf_trace_enabled

Enable or disable low-level phase timing traces for this process.

set_profile_enabled

Enable or disable the scanner’s detailed diagnostic profile collector.

set_regex_dfa_limit

Override the per-regex DFA size limit for this process. Call before scanning. 0 resets to the compiled default. Tier-A config knob (default → TOML → CLI).

Type Aliases

Result

Specialized Result type for scanning operations.