silens-scan 0.1.0

Privacy-first Rust core for detecting, querying, and safely transforming secrets and sensitive data
Documentation

Silens Scan

Privacy-first Rust core for detecting, querying and safely transforming secrets and sensitive data.

Silens Scan is designed as a reusable engine rather than an application shell. Callers provide UTF-8 text and retain control of I/O, storage and presentation. The core performs detection, validation, deterministic normalization, reporting and share-safe transformations without storing matched secret values inside public findings.

The serial core is suitable for native applications and WASM/PWA integrations. Native callers can optionally enable Rayon-backed parallel scanning across independent inputs.

Capabilities

  • Deterministic UTF-8 scanning
  • Literal, prefix, suffix and regex rules
  • Capture-aware regex projection
  • Deterministic and contextual validators
  • Selectable built-in detector catalog
  • Immutable per-source ScanReport
  • Ordered batch ScanResults<K>
  • Lazy query/filter API with explicit sorting
  • Aggregated ScanSummary
  • Optional remediation guidance per finding
  • Safe redaction
  • Semantic template generation
  • Deterministic keyed pseudonymization
  • Deterministic keyed synthetic replacements
  • Share-bundle construction
  • Optional Serde support
  • Optional Rayon parallel batch scanning
  • Unicode-aware source locations
  • Stable serial/parallel ordering
  • Reusable cached built-in scanner
  • No matched secret value stored in public findings

Install

[dependencies]
silens-scan = "0.1"

Optional features are deliberately independent:

silens-scan = { version = "0.1", features = ["serde", "parallel"] }

Feature Default Purpose


serde no Serialize and deserialize public data contracts parallel no Scan independent inputs concurrently with Rayon

The default build keeps the core serial and does not pull Rayon or Serde.

Quick start

Scanning always uses a batch-oriented API. One source is simply a one-element batch.

use silens_scan::Scanner;

let scanner = Scanner::default();

let results = scanner.scan([
    ("config.env", "STRIPE_SECRET_KEY=sk_live_example"),
    ("settings.toml", "log_level = \"info\""),
]);

println!("{}", results.summary());

for (source, finding) in results.findings() {
    println!(
        "{source}:{}:{} {}",
        finding.location().line(),
        finding.location().column(),
        finding.rule_id(),
    );
}

Caller-provided keys are preserved in the result. Each ScanEntry<K> also retains the original source byte length, while Finding contains metadata and coordinates rather than a copy of the matched secret.

Detection model

The built-in scanner combines multiple matching families behind one compiled execution pipeline:

UTF-8 source
    │
    ▼
CompiledRuleSet
    │
    ├── Aho-Corasick ── literal / prefix
    ├── regex engine ── pattern / capture
    └── specialized ─── suffix
    │
    ▼
candidate validation
    │
    ├── deterministic validators
    └── contextual validators
    │
    ▼
deterministic normalization
    │
    ▼
line / Unicode-column finalization
    │
    ▼
ScanReport → ScanResults<K>

A single source is scanned serially. With parallel, independent sources are distributed through Rayon while the same immutable scanner is reused.

Built-in detection

The current built-in catalog covers detector families for:

  • GitHub
  • Stripe
  • Cloudflare
  • Slack
  • Telegram
  • JWT
  • AWS
  • Azure
  • GCP
  • password and passphrase fields
  • sensitive hashes
  • generic API keys, tokens and secrets

The canonical selectable built-in pack is exposed as builtins::CURRENT.

Custom rules can be composed with the builder:

use silens_scan::{Remediation, Rule, Scanner, Severity};

let scanner = Scanner::builder()
    .rule(
        Rule::prefix("acme.api-key", "acme_", Severity::Critical)
            .with_remediation(Remediation::RotateCredential),
    )
    .rule(
        Rule::literal("internal.marker", "PRIVATE_VALUE", Severity::High)
            .with_remediation(Remediation::RemoveSensitiveValue),
    )
    .build()?;

# Ok::<(), silens_scan::ScannerBuildError>(())

Reports, queries and summaries

ScanResults<K> is the batch result. It exposes direct iteration as well as a lazy query surface.

use silens_scan::{ScanSort, Scanner, Severity};

let scanner = Scanner::default();
let results = scanner.scan([
    ("a.env", "TOKEN=example"),
    ("b.env", "MODE=production"),
]);

let high_priority = results
    .query()
    .minimum_severity(Severity::High)
    .sort(ScanSort::Location);

for (source, finding) in high_priority.iter() {
    println!("{source}: {}", finding.rule_id());
}

let summary = results.summary();
println!("{summary}");

Queries can filter by exact or minimum severity, exact or minimum confidence, exact rule identifier, and convenience predicates such as critical/high-priority/high-confidence. Sorting is explicit and happens after lazy filtering.

ScanSummary reports:

  • sources and bytes scanned
  • reports with and without findings
  • total findings
  • counts per severity
  • clean/critical status helpers

Remediation

A finding may carry an optional Remediation. Remediation is presentation-safe guidance attached to the rule, not an automatic mutation of the source.

for (_, finding) in results.findings() {
    if let Some(remediation) = finding.remediation() {
        println!("{}: {}", remediation.label(), remediation.message());
    }
}

This allows applications to present actionable guidance such as rotating a credential, replacing a private key, changing a password or removing a sensitive value without embedding application-specific copy into the scanner pipeline.

Transformations

The transform module turns a ScanReport plus the original source into share-safe output.

Redaction

Redaction is the conservative transform. It replaces detected spans and safely merges overlaps.

use silens_scan::{Rule, Scanner, Severity, transform::redact};

let scanner = Scanner::builder()
    .rule(Rule::literal("credential", "SECRET", Severity::High))
    .build()?;

let source = "TOKEN=SECRET";
let results = scanner.scan([("memory", source)]);
let report = results.single_report().expect("one report");

assert_eq!(redact(source, report)?, "TOKEN=[REDACTED]");

# Ok::<(), Box<dyn std::error::Error>>(())

Use redact_with when a custom replacement marker is required.

Semantic templates

Templates preserve the semantic rule identity while discarding the matched value:

TOKEN=<SILENS:credential>

TemplateOptions can configure the namespace and deterministic per-rule numbering. Template generation rejects ambiguous overlapping findings instead of guessing which semantic placeholder should win.

Pseudonymization

pseudonymize replaces findings with deterministic keyed pseudonyms. The same input value under the same key produces the same pseudonym, allowing correlation without retaining the original value.

use silens_scan::transform::PseudonymizationOptions;

let options = PseudonymizationOptions::new([0x31; 32]);

The key is supplied by the caller and is not part of a public finding.

Synthetic replacements

synthesize creates deterministic keyed synthetic values. For supported detector families it preserves useful structural characteristics while intentionally producing validator-invalid output, so generated material can be used safely in examples and shareable artifacts.

use silens_scan::transform::SynthesisOptions;

let options = SynthesisOptions::new([0x53; 32]);

Share bundles

ShareBundle applies one explicit transformation mode to a batch while preserving source keys and a manifest containing the transformation mode, summary and generation time.

Available modes are represented by ShareMode; the manifest exposes the non-secret ShareModeKind.

Use share bundles when an application needs to package transformed sources and scan metadata together for export, support or collaboration workflows.

Canonical examples and golden corpus

The repository contains a canonical fixture corpus under:

examples/fixtures/
├── inputs/
└── outputs/
    ├── reports/
    ├── redacted/
    ├── templates/
    ├── pseudonymized/
    └── synthesized/

These are not hand-written expected outputs. examples/generate_fixtures.rs runs the real scanner and transformation APIs to regenerate them:

cargo run --example generate_fixtures --features serde

Golden tests independently generate the same artifacts in memory and compare them with the committed outputs. This keeps documentation examples tied to actual crate behavior.

The shared corpus scanner intentionally exercises custom prefix, regex and literal rules plus remediation metadata.

Parallel scanning

Enable parallel to scan independent sources concurrently:

silens-scan = { version = "0.1", features = ["parallel"] }
let results = scanner.parallel_scan([
    ("config.env", env_source),
    ("settings.toml", toml_source),
    ("service.json", json_source),
]);

Serial and parallel execution use the same per-source pipeline and preserve input order. Silens Scan does not split one source into chunks and does not create a private thread pool; Rayon uses the current pool.

Parallelism is therefore an application choice, not a semantic difference in scanning.

Serde

Enable serde when results need to cross an application boundary or be persisted/serialized:

silens-scan = { version = "0.1", features = ["serde"] }
let json = serde_json::to_string_pretty(&results)?;

The canonical fixture generator uses this feature for JSON report artifacts.

Coordinate contract

Locations have an explicit mixed coordinate contract:

  • start: zero-based UTF-8 byte offset, inclusive
  • end: zero-based UTF-8 byte offset, exclusive
  • line: one-based
  • column: one-based Unicode scalar position

Byte offsets are suitable for exact source slicing. Line/column values are presentation-oriented.

Silens Scan does not copy the matched source value into a public Finding.

Privacy and trust boundary

Silens Scan is a core library. It does not decide where source data comes from or where transformed output is sent.

The intended boundary is:

application owns I/O
        │
        ▼
     &str input
        │
        ▼
   Silens Scan
        │
        ├── metadata-only findings
        └── explicit transformations

This makes the serial core suitable for local-first WASM/PWA and desktop use: applications can keep scanning and transformation local to the process/browser. Network access, remote repositories, uploads, authentication and persistence belong to the integrating application rather than this crate.

Performance

Criterion benchmarks below were measured with Rust 1.97.1. They are workload- and machine-specific engineering baselines, not portable performance guarantees.

Core scanning

Workload Median time Throughput


Built-ins, 1 KiB 20.25 µs 66.81 MiB/s Built-ins, 64 KiB 253.90 µs 247.64 MiB/s Built-ins, 1 MiB 3.759 ms 266.13 MiB/s 64 custom literal rules 48.09 µs 1.27 GiB/s 512 custom literal rules 48.93 µs 1.25 GiB/s 64 custom mixed rules 151.32 µs 413.04 MiB/s Full built-in pipeline 253.94 µs 246.12 MiB/s

Match density

For a 64 KiB input:

Density Findings Median time Throughput


None 0 211.97 µs 294.85 MiB/s Sparse 7 ~255 µs ~246--248 MiB/s Dense 1496 ~2.85 ms ~22 MiB/s

The dense fixture intentionally produces roughly one final finding every 43 bytes, making result production and validation a substantial part of the workload.

Parallel batch scanning

The benchmark scans 32 independent sources.

Source size Serial Parallel Approx. speedup


4 KiB × 32 1.027 ms 244.49 µs 4.20× 64 KiB × 32 8.161 ms 1.307 ms 6.24× 1 MiB × 32 121.16 ms 18.24 ms 6.64×

Parallel throughput reaches approximately:

  • 563 MiB/s for 4 KiB sources
  • 1.50 GiB/s for 64 KiB sources
  • 1.71 GiB/s for 1 MiB sources

Build cost

Ruleset Median build time


Built-in pack, cold compile 49.69 ms 4 custom literal rules 15.58 µs 64 custom literal rules 60.95 µs 512 custom literal rules 299.24 µs

The default built-in scanner uses a shared compiled cache, so the cold built-in compile cost is paid once per process and reused afterward.

Performance policy

The 0.1.0 engine is performance-frozen unless a correctness issue is found. Manual SIMD has deliberately not been introduced: current workloads do not justify the extra architecture-specific complexity. It can be reconsidered if real-world profiling exposes a concrete bottleneck.

Testing and hardening

The release test surface includes:

  • unit tests across rules, validators, reporting and transformations
  • doctests for public APIs
  • canonical end-to-end fixtures
  • golden output verification
  • serial/parallel equivalence
  • deterministic transformation checks
  • UTF-8 scalar-column checks
  • CRLF preservation
  • empty-input behavior
  • dense repeated-finding stress
  • large single-line input
  • Serde round-trip coverage

The canonical corpus is intended to remain part of the public documentation as the crate evolves.

Scope

Silens Scan intentionally stops at the reusable scanning core.

Application-level concerns such as file selection, remote repository loading, authenticated workflows, uploads, UI, persistence and subscription features belong to consumers such as the Silens Scan web application and Silens Studio.

A standalone CLI is not required by the core architecture and is not part of the 0.1.0 scope.

Release status

0.1.0 targets the first public crates.io release.

The core capability set is frozen. Remaining release work is documentation, example polish, package metadata and publication validation.

License

MIT