Crate threatflux_string_analysis

Expand description

§ThreatFlux String Analysis

An in-memory Rust library for tracking strings across files, enriching them with context, and applying configurable categorization, retention, and analysis heuristics.

ThreatFlux String Analysis is designed for binary-analysis, forensic, and security-pipeline enrichment. Its indicators are evidence for an application to interpret; they are not malware verdicts, reputation data, or a replacement for validation by a security analyst.

§Highlights

Track occurrences, source files, timestamps, and discovery context
Categorize URLs, paths, registry keys, commands, libraries, and other strings
Calculate byte-level Shannon entropy
Apply built-in or application-defined regular-expression patterns
Filter statistics by occurrence count, length, category, file, hash, time, entropy, suspicion, or regular expression
Search tracked values and rank related strings with a documented heuristic
Bound retained strings, source identities, occurrence detail, categories, indicators, and input byte lengths through AnalysisConfig
Share tracker state safely between clones

§Install

[dependencies]
threatflux-string-analysis = "0.2.1"

Version 0.2.1 requires Rust 1.95.0 or newer.

§Quick start

use threatflux_string_analysis::{StringContext, StringTracker};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let tracker = StringTracker::new();

    tracker.track_string(
        "powershell.exe -EncodedCommand ...",
        "/samples/example.bin",
        "sha256:example",
        "example-scanner",
        StringContext::Command {
            command_type: "PowerShell".to_owned(),
        },
    )?;

    let statistics = tracker.get_statistics(None)?;
    println!("tracked strings: {}", statistics.total_unique_strings);
    println!(
        "heuristic-positive strings: {}",
        statistics.total_suspicious_strings
    );

    Ok(())
}

The default tracker applies the built-in categorizer and pattern set. A URL or IP-address match is informational by default; command, credential, malware, and other explicitly suspicious patterns may contribute a heuristic signal.

§Configure retention

Configuration is validated when a custom tracker is constructed:

use threatflux_string_analysis::{AnalysisConfig, StringTracker};

fn configured_tracker() -> Result<StringTracker, Box<dyn std::error::Error>> {
    let config = AnalysisConfig {
        max_occurrences_per_string: 128,
        ..AnalysisConfig::default()
    };

    Ok(StringTracker::with_config(config)?)
}

Invalid limits and non-finite thresholds are rejected. At unique-string capacity, repeated values remain accepted but a new distinct value returns CapacityExceeded; the tracker does not silently evict an existing entry. Per-string occurrence detail retains the newest ingested records, while the aggregate count continues to describe every accepted observation. See the behavior contract for the complete semantics.

§Filter statistics

use threatflux_string_analysis::{StringFilter, StringTracker};

fn suspicious_commands(
    tracker: &StringTracker,
) -> Result<u64, Box<dyn std::error::Error>> {
    let filter = StringFilter {
        categories: Some(vec!["command".to_owned()]),
        suspicious_only: Some(true),
        ..StringFilter::default()
    };

    Ok(tracker.get_statistics(Some(&filter))?.total_unique_strings)
}

Every populated filter field participates in the query. Malformed regular expressions return an error instead of silently broadening the result.

§Custom analysis

The crate exposes three extension points:

StringAnalyzer computes entropy and heuristic indicators.
Categorizer assigns one or more descriptive categories.
PatternProvider manages compiled pattern definitions.

Use StringTracker::with_components for application-specific defaults or StringTracker::with_components_and_config for custom components and limits. Read the pattern guide before treating a custom match as security-relevant.

§Behavioral boundaries

The tracker is in-memory only; it does not persist or transmit observations.
Cloned trackers share the same retained entries.
Values are limited to 1 MiB of UTF-8 by default. Each path, hash, tool name, and owned context field is limited to 16 KiB by default; oversize input is rejected before mutation.
Count and field limits are independent ceilings, not a single heap-byte budget. Choose them together for the deployment’s memory envelope; maximum values can multiply into a large retained data set.
Entropy is calculated over UTF-8 bytes, so it is not a language model or a reliable encrypted-content detector.
Categories and indicators are heuristic and can produce false positives and false negatives.
Custom analyzers and categorizers are trusted in-process code. Their panics propagate to the caller, although callbacks run outside the tracker lock.
Related-string scores are ranking hints, not probabilistic confidence values.
Statistics expose bounded sample lists; category distributions can still contain one key per retained category. Use targeted filters or lookup methods when an application needs a specific entry.

See the behavior contract for filtering, ordering, retention, timestamp, concurrency, and error semantics.

§Examples and guides

Basic usage — tracking and statistics
Custom patterns — domain-specific patterns
Security-log analysis — extracting and correlating log artifacts
Pattern guide — pattern and indicator semantics
File-scanner integration — integration boundaries
Migrating to 0.2 — 0.1 upgrade guide

§Development and security

Contributing — contribution workflow
Development — local setup and commands
Testing — validation matrix
Security policy — private vulnerability reporting
Changelog — release history

§License

Licensed under the MIT License.

Structs§

AnalysisConfig: Configuration for analysis and bounded in-memory tracking.
CategoryRule: Rule for categorizing strings.
DefaultCategorizer: Default heuristic categorizer.
DefaultPatternProvider: Built-in pattern provider.
DefaultStringAnalyzer: Default byte-entropy and regex-based analyzer.
FileIdentity: Stable identity of a source file.
Pattern: A validated, compiled pattern used for analysis and categorization.
PatternDef: Serializable pattern definition.
StringAnalysis: Result of analyzing one string value.
StringCategory: A named category that can be assigned to a string.
StringEntry: Aggregate information retained for one distinct string value.
StringFilter: Filter criteria applied to statistics.
StringOccurrence: Record of one string occurrence.
StringStatistics: Statistics about a filtered tracker snapshot.
StringTracker: Thread-safe, bounded in-memory string tracker.
SuspiciousIndicator: Explainable evidence that contributed to a suspicious result.

Enums§

AnalysisError: Errors returned by validation, analysis configuration, and bounded tracking.
StringContext: Context in which a string was observed.

Constants§

DEFAULT_MAX_CATEGORIES_PER_STRING: Default number of aggregate category names retained for each value.
DEFAULT_MAX_INDICATORS_PER_STRING: Default number of suspicious indicators retained for each value.
DEFAULT_MAX_INPUT_BYTES: Default maximum UTF-8 byte length of an analyzed value.
DEFAULT_MAX_OCCURRENCES_PER_STRING: Default number of detailed occurrences retained for each distinct value.
DEFAULT_MAX_SOURCE_BYTES: Default maximum UTF-8 byte length of each source or context field.
DEFAULT_MAX_UNIQUE_FILE_IDENTITIES_PER_STRING: Default number of distinct file identities retained for each value.
DEFAULT_MAX_UNIQUE_STRINGS: Default number of distinct string values retained by a tracker.
DEFAULT_SUSPICIOUS_ENTROPY: Default entropy threshold used to emit a high-entropy indicator.
VERSION: Version of the crate selected at compile time.

Traits§

Categorizer: Interface for deterministic, thread-safe string categorizers.
PatternProvider: Provider interface for validated analysis patterns.
StringAnalyzer: Interface for deterministic, thread-safe string analyzers.

Type Aliases§

AnalysisResult: Result type for string analysis operations.
CategoryMatcher: Thread-safe predicate used by a CategoryRule.
DateTimeRange: Inclusive range applied to an entry’s first_seen timestamp.