Crate keyhog_core

Expand description

§keyhog-core

Shared domain types for the KeyHog secret scanner: the detector spec model, the embedded detector corpus, chunk and finding types, redaction, and the report-safe boundary every other crate projects through.

Part of the KeyHog secret scanner.

use keyhog_core::{detector_spec_by_id, embedded_detector_count, redact};

// The compiled-in corpus. Every crate borrows this one materialized copy.
let count = embedded_detector_count();
assert!(count > 0);

// Detectors are addressable by their stable id. `aws-access-key` is in the
// shipped corpus, so this resolving is itself a check that the corpus loaded.
let spec = detector_spec_by_id("aws-access-key").expect("shipped detector");
assert_eq!(spec.id, "aws-access-key");

// Anything derived from scanned bytes is redacted before it can be printed.
// The control comes first: prove the needle is visible in the input, so a
// probe that can see nothing at all cannot pass this as a clean redaction.
let plaintext = "AKIAIOSFODNN7EXAMPLE";
assert!(plaintext.contains("IOSFODNN7"));
let safe = redact(plaintext);
assert!(!safe.contains("IOSFODNN7"));

§Public entry points

load_embedded_detectors_or_fail parses the compiled-in corpus and returns a typed error naming every detector that failed. It never returns a partial set.
embedded_detector_specs is the one materialized corpus. Borrow from it rather than re-parsing, and detector_spec_by_id indexes it.
embedded_detector_count, detector_digest, and git_hash identify exactly which corpus and build you are running. Benchmarks and caches key on these.
Credential and SensitiveString hold plaintext. redact and the report types in report are the projection you use before anything leaves the process.
spec owns detector validation. A detector that does not validate is a build or authoring error, never a scan-time condition.

§Failure behavior

A corrupt compiled-in corpus stops startup with the exact parse error. The crate does not substitute an empty or reduced detector set, because a scanner that silently loads fewer detectors reports a clean result on a file that has a secret in it.

§Features

The default build has no optional features and no native build prerequisite. The detector corpus is compiled in by build.rs from the workspace detectors/ directory, so the crate needs no data files at runtime.

§Documentation

Detectors describes the spec schema these types model.
Architecture describes where this crate sits and which direction dependencies run.
API documentation is on docs.rs. Core types shared across all KeyHog crates.

Re-exports§

pub use crate::ascii_ci::contains_bytes_ignore_ascii_case;
pub use crate::ascii_ci::contains_ignore_ascii_case;
pub use crate::ascii_ci::ends_with_ignore_ascii_case;
pub use crate::ascii_ci::starts_with_ignore_ascii_case;

Modules§

ascii_ci: Shared ASCII case-insensitive primitives: starts_with / ends_with_ignore_ascii_case and the ci_find byte-substring search. Single owner (ONE PLACE) for allocation-free case-folded comparisons on the scanner and verifier hot paths, callers use these instead of allocating a lowercased copy of each candidate.
git_lfs: Git-LFS pointer recognition, shared by the scanner (oid suppression) and sources (unscanned-blob coverage gap). Git-LFS pointer recognition.
guard_state: Perpetual guard root state machine, policy identity, and receipt types. Perpetual guard root state machine, policy identity, and receipt types.
guard_store: Durable guard state store: schema, tables, and memory-bounded LRU. Durable guard state store: schema, tables, and memory-bounded LRU.
json_selector: Detector verification response selector grammar and evaluator. Bounded rooted selectors for detector-owned verification responses.
retry: The one retry policy: bounded attempts, one backoff, one classification of transient versus permanent. Retry is the second choice; see the module docs for what must never be routed through it. The one retry policy: bounded attempts, one backoff, one classification.
timing: Shared paired performance statistics used by release gates and routing evidence. Shared statistical evidence for paired performance comparisons.
verification_domain: Verification-domain policy shared by detector validation and the network verifier. Shared verification-domain policy.
winpath: Canonical Windows-path classification predicates.

Structs§

AccessTarget: One resource a credential is believed to open.
AccessTargetCoverage: What the pass actually managed to look at.
AccessTargetReport: The complete result of one association pass.
Allowlist: User-defined suppressions loaded from .keyhogignore: credential hashes, detector IDs, and path globs.
BetaCounters: A detector’s running Beta posterior counters. Always ≥1 each (Beta(1,1) uniform prior baseline) to avoid posterior_mean undefined when a detector has had no observations yet.
Calibration: Process-wide calibration store. Concurrent updates are serialized via a single RwLock because update events are rare (one per keyhog calibrate invocation or per verifier outcome) and the locked region is constant-time. We deliberately don’t shard via DashMap - the persisted artifact is small enough that contention is a non-issue.
CanonicalHexKeyMaterialSpec: One detector-local pure-hex key-material policy.
Chunk: A scannable chunk of text with metadata about where it came from.
ChunkMetadata: Metadata that tracks the source location for a scanned chunk.
CompanionSpec: Secondary pattern used to confirm a primary match or provide extra context.
ConfidenceProvenance: Exactly how a target’s confidence was produced.
CorrelatedCredential: A credential risk assembled from several findings.
CorrelatedLocation: One place a correlated credential was seen.
CorrelatedMember: One finding participating in a correlation group.
CoverageGap: One reason some findings had no file context, and how many were affected.
Credential: Opaque credential bytes. The inner Arc<Zeroizing<Box<[u8]>>> clones are cheap (refcount bump) but every owning Credential zeroizes on drop. Arc lets the engine intern identical credentials without copying; when the last ref drops, Zeroizing<Box<[u8]>> overwrites the heap allocation before Box::drop returns it to the allocator.
CredentialHash: SHA-256 digest of a credential.
CredentialShape: Per-detector credential SHAPE constraint ([detector.credential_shape]), OWNED HERE per the architecture law (was a centralized rules/detector-credential-shapes.toml [[shape]] list keyed by detector id, a per-detector property in a second file). A candidate whose byte length / prefix / post-prefix body length does not fit the declared shape is suppressed by the scanner’s shape gate (CredentialShapeRule::allows). Only a couple of fixed-format vendor detectors declare it: aws-access-key is exactly 20 bytes; anthropic-api-key is sk-ant-api03- + an 80..=120 body.
DedupedMatch: A group of related raw matches representing a single distinct secret finding.
DetectorCorpusManifest: Directory-scoped compatibility contract for detector TOML files.
DetectorDecodeTransformSpec: Literal admission prefixes for detector-owned evasion recovery.
DetectorFile: Wrapping struct for a detector TOML file.
DetectorIntrospection: Redaction-safe serialized view of one detector declaration.
DetectorMatchConfidenceSpec: Complete detector-local confidence policy for regex candidates.
DetectorMlPolicySpec: Complete detector-local configuration for the shared ML scoring engine.
DetectorPlausibilityPolicySpec: Strict candidate-plausibility policy owned by one detector TOML.
DetectorPostMatchConfidenceSpec: Detector-local confidence penalties applied after optional model scoring.
DetectorRelationSpec: A bounded relationship between findings from two detector owners.
DetectorSpec: A complete detector definition loaded from a TOML file.
DetectorTestSpec: One inline detector self-test fixture ([[detector.tests]]).
EntropyFallbackConfidenceSpec: Confidence mapping for one detector’s synthetic entropy findings.
EntropyFallbackMetadata: Detector-owned identity for a finding emitted by the entropy fallback path.
EntropyFloorBucket: One length bucket of a detector’s DetectorSpec::entropy_floor. Owned in the detector’s TOML (entropy_floor = [{ max_len = 24, floor = 3.0 }, { floor = 3.5 }]).
EntropyShapeSpec: A declarative structural shape that may cross a detector’s broad isolated entropy floor. Every field is detector TOML data and the scanner keeps one general matcher, so a new shape family (base64, hex block) is added by a detector TOML rather than a scanner enum variant.
FileContent: One file’s content prefix, plus whether it was cut short.
FilesystemContent: Reads from the local filesystem.
GenericAssignmentConfidenceSpec: Confidence mapping for a detector-owned generic assignment candidate.
HardeningReport: Outcome of a hardening attempt - collected so callers can log which protections actually took.
HeaderSpec: Custom HTTP header specification.
JsonReportCoverageGap: One scan-wide coverage gap preserved in a versioned JSON report.
JsonReportEnvelope: Versioned machine-readable JSON report.
JsonReportSchemaVersion: Version marker carried by every versioned JSON report.
JsonlStream: One validated segment of a JSONL input. Concatenated streams produce one segment per header, so boundaries remain explicit instead of being inferred from finding content.
JsonlStreamHeader: Header written as the first record of a versioned JSONL stream.
JsonlStreamSummary: Terminal record written when a versioned JSONL stream completes.
LoadedDetectorCorpus: A validated detector corpus paired with the schema identity that selected its normalization rules.
MatchLocation: Where a credential was found: file path, line number, commit, and author.
MerkleIndex: In-memory file-hash index loaded from / saved to a JSON cache file.
MerkleLoadReport: Result of loading a persisted MerkleIndex cache.
MetadataSpec: Metadata field specification for verification results.
ObservedPaths: A Vec-compatible path list that records direct mutable access.
OobSpec: Out-of-band callback verification configuration.
PatternSpec: A regex pattern with optional capture group and description.
RawMatch: A raw pattern match before verification or deduplication.
RawMatchDedupKey: Borrowed raw-match identity used before report-scope deduplication.
RedactedFinding: Redacted, disk-safe view of a RawMatch. Carries only the SHA-256 hash and a “first4…last4” preview, never the plaintext credential. Use this before verification; VerifiedFinding is the final report-safe shape.
ReportError: Common error type used by all reporters. The Error type, a wrapper around a dynamic error type.
ResolvedScanManifest: Stable, machine-diffable description of the resolved detection mode.
RuleSuppressor: Parsed .keyhogignore.toml - a list of [[suppress]] rules, each compiled into a RuleFormula.
ScanBackendRecoverySummary: Bounded, non-secret summary of one completed exact recovery.
ScanConfig: Configuration for a scan run.
ScanReport: The format-neutral input shared by every report renderer.
ScanReportMetadata: Format-neutral operator-visible metadata for a scan report.
SensitiveString: A heap-allocated string that is zeroized on drop.
ShapeGrouping: Fixed-width separator grouping, e.g. a Bluesky app password’s four dash-separated groups of four. Absent means an ungrouped run.
SourceAdmissionSpec: Positive source selectors for detector families valid only in known locations.
StateFileWriteLock: Exclusive advisory lock held across a state file’s read/merge/write cycle.
StaticRecoveryMetrics: Exact, non-secret bounded static-recovery telemetry for one scan.
StepSpec: A single step in a multi-step verification flow.
SuccessSpec: Criteria for a successful verification response.
TargetEvidence: Why one target is attributed to one credential.
TargetedLocation: Where a credential with access targets was found.
VerifiedFinding: A finding after verification - the final output.
VerifySpec: Live verification configuration for a detector.

Enums§

AccessTargetKind: What kind of thing a credential opens.
AuthSpec: Authentication scheme for verification requests.
CalibrationLoadError: Error returned when an existing calibration cache cannot be trusted.
ConfigError: Errors returned while validating a scan configuration.
ContentError: Why a file could not be turned into an index.
CorrelationKind: How a correlation group was joined.
CorrelationRole: Why one finding belongs to a correlation group.
CoverageGapReason: Why the pass could not build file context for some findings.
DedupScope: Deduplication scope for grouping findings.
DetectorBase64Alphabet: Exact base64 dialect accepted by a detector-owned offline validator.
DetectorCorpusError: Failure to compose an effective detector corpus.
DetectorCorpusMode: How a custom detector corpus participates in the effective corpus.
DetectorKind: Which scan phase produces a detector’s findings (see DetectorSpec::kind).
DetectorMlMode: How the shared ML model participates in one detector path.
DetectorRelationKind: Cross-detector evidence operation applied during deterministic match resolution.
DetectorValidatorSpec: An offline validator declared by one detector.
EntropyDetectionRole: Detector-owned entry roles for the shared entropy engine.
EntropyFallbackClass: Semantic role of a detector-owned synthetic entropy finding.
EvidenceDirection: How a contextual evidence match must be positioned relative to the primary credential.
EvidenceRequirement: Whether contextual evidence admits, strengthens, or rejects a detector match.
EvidenceScope: The structural boundary in which contextual evidence may satisfy a relation.
EvidenceValueRelation: The value relationship between a selected evidence capture and the primary credential.
HttpMethod: HTTP method for verification requests.
MerkleLoadStatus: Operator-relevant status for a Merkle cache load.
OobPolicy: How OOB observation combines with HTTP success criteria.
OobProtocol: Out-of-band callback protocol expected from a successful exfil.
ProviderEvidenceRole: Stable semantic role of provider evidence exposed in findings.
ProviderEvidenceSensitivity: Confidentiality policy for detector-owned provider evidence.
QualityIssue: Quality issue found in a detector spec.
Redaction: What was done to a target value before it was allowed into an artifact.
ReportFormat: Output format and formatter options for write_report.
RuleSuppressorError: Errors from loading or parsing .keyhogignore.toml.
ScanCompletionStatus: Terminal state carried by detached scan artifacts.
ScriptEngine: Script interpreter names accepted by the detector TOML schema.
Severity: Severity level for a finding.
ShapeCharset: Character class an isolated entropy shape admits. Replaces the former per-shape enum variant so a new shape family is TOML data, not scanner code.
SourceCoverageGapKind: Machine-readable reason a requested source surface was not fully scanned.
SourceError: Errors returned by input sources while enumerating or reading content.
SpecError: Errors returned while loading or validating detector specifications.
SuccessPolicy: How a verifier response establishes a successful credential check.
TargetRelation: How a target was tied to a credential.
VerificationResult: Result of live verification: whether the credential is active, revoked, or untested.

Constants§

DEFAULT_ENTROPY_BPE_MAX_BYTES_PER_TOKEN: Single owner of the BPE “rare-not-random” gate default (the ScanConfig::default().entropy_bpe_max_bytes_per_token knob). An entropy / generic candidate whose cl100k_base bytes-per-token is STRICTLY GREATER than this compresses into few common subword tokens, word-like (a probable false positive: dotted API paths, prose, XML) rather than a random secret and is suppressed. 2.2 is the empirical CredData F1 peak (see keyhog_scanner::entropy::bpe; the offline A/B lifted F1 0.368→0.424).
DEFAULT_ENTROPY_THRESHOLD: Single owner of the Tier-A generic-entropy gate default (the ScanConfig::default().entropy_threshold knob). The scanner’s adjudicate fallback (generic_entropy_floor) resolves an absent per-scan value to this SAME number, so it references this const rather than re-spelling 4.5: the two must stay equal by construction, not by two hand-kept literals.
DEFAULT_MAX_FILE_SIZE_BYTES: Shipped filesystem per-file scan cap.
DETECTOR_CORPUS_MANIFEST_FILE: Canonical file name for the directory-scoped detector corpus manifest.
DETECTOR_CORPUS_MAX_FORWARD_SCHEMA_VERSION: Highest newer detector schema this binary may inspect additively.
DETECTOR_CORPUS_MIN_SCHEMA_VERSION: Oldest legacy detector schema this binary can migrate deterministically.
DETECTOR_CORPUS_SCHEMA_VERSION: Detector schema authored and enforced by this binary.
DETECTOR_TOML_FILE_BYTES: Maximum accepted size for one on-disk detector TOML file.
HYPERSCAN_CACHE_FILE_BYTES: Hard cap for one serialized Hyperscan shard cache file, including the KeyHog header.
HYPERSCAN_CACHE_HEADER_LEN: Byte length of the KeyHog Hyperscan cache header: magic plus little-endian version.
HYPERSCAN_CACHE_MAGIC: Magic bytes at the front of every KeyHog Hyperscan shard cache file.
HYPERSCAN_CACHE_VERSION: KeyHog-owned cache header version for serialized Hyperscan shard files.
JSONL_REPORT_SCHEMA_MINOR: Current minor version for the versioned JSONL stream contract.
JSON_REPORT_SCHEMA_MAJOR: Current major version for the versioned JSON report envelope.
JSON_REPORT_SCHEMA_MINOR: Current minor version for the versioned JSON report envelope. Minor 9 adds the optional correlations array and minor 10 the optional access_targets object, each absent unless the producer opted in.
KEYHOG_MATCHER_ARTIFACTS_SUBDIR: Sibling of [KEYHOG_CACHE_SUBDIR] used for MatcherArtifact .khm files.
MATCHER_ARTIFACT_FILENAME_PREFIX: Filename prefix for MatcherArtifact cache files (matcher-<hex>.khm).
MATCHER_ARTIFACT_FORMAT_VERSION: MatcherArtifact on-disk envelope version shared with lockdown header checks.
MATCHER_ARTIFACT_MAGIC: On-disk magic for MatcherArtifact cache files (KHMA).
MATCHER_ARTIFACT_SUFFIX: Filename suffix for MatcherArtifact cache files.
STATIC_RECOVERY_METRICS_SCHEMA_VERSION: Schema generation for exact bounded static-recovery telemetry.

Traits§

FileContentSource: Where the association pass gets file bytes.
Source: Produces chunks of text for the scanner to process. Each implementation handles a different input source.

Functions§

access_target_rule_ids: Rule ids the shipped policy defines, in file order.
apply_protections: Apply the process protections for the requested security mode.
apply_protections_with_persistence_paths: Apply process protections and, in lockdown mode, fail closed on known persistence artifacts outside the default keyhog cache root.
associate_access_targets: Attach access targets to a finding set, reading file context from disk.
associate_access_targets_with: Attach access targets using a caller-supplied content source.
calibration_default_cache_path: Default calibration cache location: $XDG_CACHE_HOME/keyhog/calibration.json (or the macOS/Windows equivalents via the dirs crate).
compose_detector_corpus: Compose an effective detector corpus without implicit merging.
compute_detector_corpus_digest: Compute a deterministic digest of current-schema detector specs.
compute_detector_corpus_digest_for_schema: Compute a deterministic, schema-bound digest of a complete effective detector corpus.
compute_spec_hash: Compute a stable BLAKE3 digest over the canonical detector set so a later scan can detect that detectors changed.
correlate_findings: Correlate the findings a report is about to publish.
correlation_composite_part_ids: Every detector id any composite row names, sorted and deduplicated.
current_executable_sha256: SHA-256 hex digest of the currently running executable, memoized once per process.
decode_standard_base64: Decode standard-alphabet base64 (with optional = padding).
dedup_cross_detector: Cross-detector dedup at emit time.
dedup_matches: Deduplicate raw matches according to the given DedupScope.
detector_digest: Effective digest identifying the EXACT embedded detector set and the directory-scoped corpus.toml schema contract compiled into this binary (<detector-count>-<fnv1a_hex>). Binding the manifest ensures caches, benchmarks, and autoroute evidence invalidate when parsing compatibility semantics change even if detector TOML bytes do not. Stamped by build.rs via cargo:rustc-env=KEYHOG_DETECTOR_DIGEST, this is the authoritative answer to “which effective corpus was compiled in” when cargo’s rerun-if-changed cannot be trusted across in-place TOML edits.
detector_spec_by_id: Canonical id → DetectorSpec lookup over the embedded corpus, built EXACTLY ONCE.
embedded_detector_count: Number of embedded detector specs (authoritative for banners and tests).
embedded_detector_specs: Every embedded detector spec, parsed EXACTLY ONCE for the whole process.
finding_metadata: Build the offline metadata for an AWS-access-key finding: always { "account_id": "<12 digits>" } for a decodable AKIA…/ASIA… key, plus { "is_canary": "true", "canary_message": <note> } when the decoded account belongs to a known canary issuer. None when credential is not a well-formed AWS access-key ID.
git_hash: Git commit SHA the binary was built from, or "unknown" for a build with no reachable .git tree (e.g. a cargo package / crates.io build). Stamped by build.rs via cargo:rustc-env=GIT_HASH; env! resolves here because the rustc-env applies to THIS crate’s compilation. Surfaced in keyhog --version and meant to be embedded in every result so a scan traces back to an exact commit (MC-06: the false “F1 regression” was a stale binary benched against HEAD, undetectable while every build reported the same empty version).
hex_encode: Lower-case hex of digest bytes. The only place the hex string is materialized for CredentialHash values (reporters, Debug).
hyperscan_cache_filename: Build the on-disk filename of a KeyHog Hyperscan shard cache file from its content shard_key: hs-<shard_key>.db. Single owner of the name FORMAT, shared by the scanner shard writer (which persists the file) and the hardening lockdown gate (which recognises/strips it via [HYPERSCAN_CACHE_PREFIX]/[HYPERSCAN_CACHE_SUFFIX]), so writer and reader can never disagree on the shard filename. Previously the writer re-inlined the hs-/.db affixes in a format!, a latent drift from this owner.
hyperscan_cache_header_is_valid: Return true when header is exactly the current KeyHog Hyperscan cache header.
key_id_canary_status: Checked canary classifier used by live-verification paths.
keyhog_matcher_artifacts_root: Absolute path of the MatcherArtifact cache root (<os-cache>/keyhog-matcher-artifacts), or None when the platform exposes no cache directory.
load_detector_corpus: Load all detector specs together with their normalized corpus schema identity.
load_detectors: Load all detector specs from a directory of TOML files. Runs the quality gate on each detector and fails closed if any detector cannot be read, parsed, or accepted by the gate.
load_embedded_detectors_or_fail: Parse the embedded detector corpus, FAILING CLOSED on any malformed TOML.
max_decode_depth_limit: Maximum recursive decode passes accepted from CLI and TOML config.
merkle_default_cache_path: Default Merkle index location: $XDG_CACHE_HOME/keyhog/merkle.idx or ~/.cache/keyhog/merkle.idx on Linux, ~/Library/Caches/keyhog/... on macOS.
parse_canary_account_ids: Parse and validate raw 12-digit account IDs from .keyhog.toml config.
parse_jsonl_stream: Parse one or more concatenated, versioned JSONL streams.
read_detector_toml_file: Read one detector TOML without allowing metadata races to bypass the cap.
redact: Redact a sensitive credential string for safe display.
redact_companions: Redact every companion value at the process boundary.
resolve_safe_bin: Resolve name to an absolute path inside one of the trusted system binary directories. Returns None if not found in any trusted dir (do NOT fall back to Command::new(name) - that’s exactly the bug).
set_extra_canary_accounts: Replace the process-local extra canary account set supplied by .keyhog.toml.
set_extra_trusted_dirs: Replace the caller-configured trusted binary directories.
sha256_hash: SHA-256 of a string as the CredentialHash domain type. This is the single source for credential hashing across the workspace (scanner, dedup, telemetry); hex encoding is a separate step at the serde/reporter boundary via hex_encode, keeping the pre-dedup hot path zero-heap.
state_file_lock_path: Canonical sibling lock filename for a KeyHog state artifact.
strip_windows_verbatim_prefix: Strip the Windows verbatim path prefix (\\?\) from a display string.
validate_access_target_policy: Validate a candidate access-target policy document without installing it.
validate_canary_accounts: Validate the compiled Tier-B canary account baseline.
validate_correlation_policy: Validate a candidate correlation policy document.
validate_detector: Validate a detector spec against the quality gate.
write_csv_coverage_report: Write a CSV scan artifact with a self-describing scan-status preamble.
write_hyperscan_cache_header: Append the current KeyHog Hyperscan cache header to a serialized-cache buffer.
write_report: Write a complete findings report in the requested format.
write_scan_report: Write a complete report from the shared scan model.

Type Aliases§

CompanionMap: Companion values keyed by scanner-compiled, reference-counted names.
HtmlScanMetadata: Compatibility name for callers that used the original HTML-only type.