rsigma-eval

rsigma-eval is an evaluator for Sigma detection rules. It compiles Sigma rules into optimized in-memory matchers and evaluates them against JSON events. Rules are compiled once; evaluation is zero-allocation on the hot path.

This library is part of rsigma.

Public API

Engine

Method	Description
`Engine::new()`	Create an empty engine
`Engine::new_with_pipeline(pipeline)`	Create engine with an initial pipeline
`set_include_event(include: bool)`	Global override: include full event JSON in all match results
`add_pipeline(pipeline)`	Add a pipeline (sorted by `priority` after add)
`add_rule(rule: &SigmaRule)`	Apply pipelines, compile, and incrementally fold into engine indexes (amortized O(1) per rule)
`add_rules(rules)`	Batched add of many rules; returns per-rule compile errors as `(index, error)` pairs and rebuilds engine indexes once
`add_collection(collection: &SigmaCollection)`	Add all rules, then apply all filters
`add_collection_with_pipelines(collection, pipelines)`	Temporarily replace pipelines, add collection, restore
`add_compiled_rule(rule: CompiledRule)`	Add a pre-compiled rule directly; folds into engine indexes incrementally (amortized O(1) per rule)
`extend_compiled_rules(rules)`	Batched add of pre-compiled rules; rebuilds engine indexes once
`apply_filter(filter: &FilterRule)`	Inject filter as `AND NOT` into referenced rules

The free function apply_filters(collection: &SigmaCollection) -> Vec<SigmaRule> merges each filter into the detection rules it targets at the AST level, as pySigma does when a collection loads, so processing pipelines applied afterwards transform the filter's fields too. rsigma-convert uses it before converting. | evaluate(event: &Event) | Evaluate all rules against an event | | evaluate_with_logsource(event, logsource) | Evaluate with subset logsource pre-filtering | | evaluate_pruned(event, logsource) | Evaluate with a caller-resolved logsource for conflict-based pruning (used by SchemaRouter) | | evaluate_batch(events: &[&Event]) | Evaluate all rules against multiple events (parallel with parallel feature) | | set_bloom_prefilter(enabled: bool) | Enable opt-in bloom-filter pre-filtering of positive substring matchers (off by default; see Bloom Pre-Filter) | | bloom_prefilter_enabled() | Whether bloom pre-filtering is currently enabled | | set_bloom_max_bytes(max_bytes: usize) | Override the bloom memory budget (default: 1 MB) | | bloom_max_bytes() | Configured bloom memory budget, or None if using the default | | set_cross_rule_ac(enabled: bool) | Enable opt-in cross-rule Aho-Corasick pre-filter (requires daachorse-index feature; see Cross-Rule AC Index) | | cross_rule_ac_enabled() | Whether the cross-rule AC pre-filter is currently enabled (daachorse-index only) | | rule_count() | Number of loaded rules | | rules() | Access the compiled rules slice | | transform_rule(rule: &SigmaRule) | Apply the engine's pipelines and return the rewritten rule as a TransformedRule, without compiling or loading (inspection only) | | transform_collection(collection: &SigmaCollection) | Per-rule transform_rule over a collection, in collection order |

Correlation Engine

Method	Description
`CorrelationEngine::new(config)`	Create with a `CorrelationConfig`
`set_include_event(include: bool)`	Global override for event inclusion
`add_collection(collection)`	Add rules and correlations
`add_rule(rule: &SigmaRule)`	Add a single detection rule
`add_correlation(corr: &CorrelationRule)`	Add a single correlation rule
`process_event(event: &Event)`	Evaluate + update correlation state (wall-clock time)
`process_event_at(event, timestamp_secs)`	Evaluate + update state with explicit timestamp
`evaluate(event: &Event)`	Run detection only (no correlation state update)
`process_with_detections(event, detections, ts)`	Feed pre-computed detections into correlation state. Rules without an `id` are matched by title, so name-only rules need unique titles here; `process_event_at` and `process_batch` keep the compiled identity
`process_batch(events: &[&Event])`	Parallel detection + sequential correlation for a batch of events
`evict_expired(now)`	Manually evict expired state entries
`introspect()` / `introspect_filtered(id, group)`	Read-only `CorrelationStateSnapshot`: per correlation and group, the current aggregate vs threshold, window contents, last alert and remaining suppression, and seconds to eviction
`state_count()`	Number of active correlation state entries
`event_buffer_count()`	Total events stored across all buffers
`event_buffer_bytes()`	Total bytes of compressed event data
`set_bloom_prefilter(enabled: bool)`	Forward to the inner detection engine (off by default)
`set_bloom_max_bytes(max_bytes: usize)`	Forward to the inner detection engine
`set_cross_rule_ac(enabled: bool)`	Forward to the inner detection engine (requires `daachorse-index` feature)

Compilation

Function	Description
`compile_rule(rule: &SigmaRule)`	Compile a parsed rule into a `CompiledRule`
`compile_detection(detection: &Detection)`	Compile a detection tree
`evaluate_rule(rule: &CompiledRule, event: &Event)`	Evaluate one compiled rule
`eval_condition(expr, detections, event, matched)`	Evaluate a condition expression tree

Explain (`explain` module)

A non-short-circuiting, bloom-free recording evaluator that backs rsigma engine explain. It visits every condition branch and field so a non-match can be explained, recording per-node and per-leaf outcomes and reasons. The trace verdict is computed from the same eval primitives the engine uses, so it can never disagree with evaluate_rule.

Function / type	Description
`explain_rule(rule: &CompiledRule, event: &Event)`	Return a `RuleExplanation` (verdict + condition tree) for one rule and one event
`RuleExplanation` / `ConditionTrace` / `DetectionTrace` / `ItemTrace` / `ArrayMemberTrace`	The serializable trace model (mirrors `ConditionExpr` / `CompiledDetection` / `CompiledDetectionItem`). `DetectionTrace` includes `AllOf`, `AnyOf`, `And`, `Keywords`, `ArrayMatch` (per-member traces, optional truncation), `Conditional` (extended array-body or top-level condition tree), and `Other` as a last-resort fallback
`ArrayEmptyReason`	Why an `ArrayMatch` node had zero members: `missing_or_null` or `empty_array`
`MatchReason`	Per-leaf reason: `Matched`, `FieldAbsent`, `ValueMismatch`, `CaseMismatch`, `Existence`, `NoKeywordMatch`

Pipeline

Function	Description
`parse_pipeline(yaml: &str)`	Parse a pipeline from a YAML string
`parse_pipeline_file(path: &Path)`	Parse a pipeline from a YAML file
`parse_sources_file(path: &Path)`	Parse a standalone source YAML file (top-level `sources:` list)
`parse_sources_dir(dir: &Path)`	Parse all `.yml`/`.yaml` source files in a directory
`validate_source_refs(pipeline, external_ids)`	Validate that `${source.*}` references resolve against declared and/or external source IDs
`apply_pipelines(pipelines, rule)`	Apply all pipelines to a rule, in slice order
`apply_pipelines_with_state(pipelines, rule)`	Same ordering, and returns the merged `PipelineState` (for backends)
`merge_pipelines(pipelines)`	Sort a pipeline slice by `priority` in place, lower first. Despite the name it does not combine or apply them; call it before the `apply_` / `transform_` functions, which walk the slice as given
`transform_rule(pipelines, rule)`	Apply pipelines to a clone and return a `TransformedRule` (rewritten rule + sorted applied ids + merged state); the input rule is untouched. Slice order, like `apply_pipelines`: sort with `merge_pipelines` to match engine behavior
`transform_collection(pipelines, collection)`	Per-rule `transform_rule` over a collection's detection rules, in collection order

TransformedRule { rule, applied_items, state } is for callers that need the rewritten Sigma AST rather than an evaluation: deriving which log channels to collect from a change_logsource-rewritten logsource, reporting the conditions a pipeline injected, or asserting in a test that a mapping applied. Loading rules into an Engine keeps only the compiled form, so this is how the rewritten rule is read. It is inspection-only and belongs on the rule-set load path, not the per-event path; when only the rewritten logsource matters, Engine::rules() already carries it after loading and costs nothing extra.

Rule field extraction (`fields` module)

Shared between rsigma rule fields and the field-observability surfaces in both engine eval and engine daemon, so the offline view matches what the engine references at runtime.

Type / function	Description
`RuleFieldSet::collect(collection, pipelines, include_filters)`	Walk a `SigmaCollection` (after optional pipeline transformations) and return every field name referenced by detection items, correlation `group-by`/threshold/alias fields, filter detections, and rule-level `fields:` metadata
`RuleFieldSet::contains(name)` / `origin(name)` / `iter()` / `names()` / `len()`	Query the resulting set; entries are sorted lexicographically
`FieldOrigin { rule_titles, sources }`	Per-field provenance: which rules touched it and from which kind
`FieldSource::{Detection, Correlation, Filter, Metadata}`	Tagged source kind, with `as_str()` for stable JSON serialization

Field observability (`field_observer` module)

Opt-in counter that records every observed field name and surfaces gap and broken-coverage signals. Consumed by both engine eval (one-shot report at end-of-run) and engine daemon (live /api/v1/fields* endpoints).

Type / function	Description
`FieldObserver::new(max_keys)`	Construct an observer with a hard cap on distinct keys (overflow drops are counted, existing keys keep updating)
`FieldObserver::observe(event: &impl Event)`	Walk `event.field_keys()` and bump per-field counters; `&self` so callers can share an `Arc`
`FieldObserver::snapshot()`	`FieldObservation { entries, events_observed, unique_keys, overflow_dropped, lifetime_events_observed, lifetime_overflow_dropped, max_keys, uptime_seconds }`. Entries sorted by descending count then name. Keys held as `Arc<str>` so the snapshot is refcount-cheap
`FieldObservation::coverage(&RuleFieldSet)`	Partition a snapshot into `FieldCoverage { unknown, intersection_count, missing }` against a rule field set in one pass. Both the daemon HTTP handlers and the eval report consume this, so the partition semantics cannot drift
`FieldObserver::reset()`	Clear counters; lifetime totals survive so Prometheus counter bridges stay monotonic

Rule drafting (`rule_draft` module)

Turns exemplar events into complete draft Sigma detection and temporal correlation rules, verified end-to-end through the real parse/compile/evaluate path. Backs rsigma rule draft.

Type / function	Description
`draft_rule(exemplars, baseline, &DraftConfig)`	Profile the exemplars, drop volatile fields (timestamps, GUIDs, counters, high-entropy uniques), score by stability times baseline rarity, infer value forms and modifiers (equals, OR list, `endswith`/`startswith`/`contains` with Sigma wildcard escaping), infer the logsource via `SchemaClassifier`, and emit + verify the rule (every exemplar must match; bounded relaxation with a minimum-field floor)
`DraftConfig`	Tunables: `max_fields`, `min_fields`, `min_prevalence`, `max_value_cardinality`, `min_token_len`, `max_baseline_token_prevalence`, include/exclude fields, title/logsource overrides, `rule_id` (caller-supplied; the core is deterministic and never generates one), `date`, `evaluate_baseline`
`DraftReport`	`rule_yaml` (parse- and lint-checked), ranked `fields` (`DraftFieldReport` with score, `Stability` class, chosen modifier, values, baseline prevalence), `exemplar_matched`, `baseline_hits`/`baseline_hit_rate`, warnings
`rule_draft::correlation::draft_correlation(groups, negatives, baseline, &CorrelationDraftConfig)`	Validate grouped timed events, infer recurring slots, grouping entity, order, and window, emit named detection documents plus a temporal correlation, and verify every group in an isolated `CorrelationEngine`
`CorrelationDraftReport`	Multi-document YAML, correlation type, group-by fields, chosen timespan, observed spans/gaps, per-slot support and selected forms, isolated verification rows, warnings
`DraftError`	`NoExemplars`, `NoCandidateFields`, `CannotMatchExemplars` (the floor error: an over-broad draft is refused, not emitted), `ForcedFieldMismatch` (a forced include-field absent from some exemplars is named, never silently dropped)

Rule exemplars (`exemplar` module)

Replays embedded rsigma.exemplars through the production Engine / CorrelationEngine collection-loading APIs. Backs rsigma rule test and the MCP test_exemplars tool.

Type / function	Description
`run_exemplars(collection, pipelines)`	Fresh engine state per exemplar; detection targets isolate to the host rule plus filters; correlations load the full collection and replay `process_event_at` from a fixed base timestamp
`ExemplarReport`	Per-exemplar pass/fail plus rules with no exemplars (for `--fail-on-missing`)
`ExemplarRunError`	Shape, ambiguous title, missing correlation reference, or compile failure

Rule tuning (`rule_tune` module)

Contrasts false-positive events with required true-positive exemplars and emits a standard Sigma filter rule only after closed before/after verification through the real engine.

Type / function	Description
`tune_rule(rule, false_positives, true_positives, &TuneConfig)`	Profile FP-stable fields, reject forms that match protected TPs, derive supported clusters when needed, emit a filter targeting the rule id, and verify the result through `Engine::add_collection`
`TuneConfig`	Bounds minimum and maximum fields, exact-value cardinality, token length, minimum cluster support, maximum clusters, partial coverage, and caller-supplied metadata
`TuneReport`	Paste-ready filter YAML, ranked field rationale, emitted selections, FP coverage, warnings, and before/after verification
`TuneError`	Empty corpora, non-firing labels, no candidate fields, no clean separator, or an internal emitted-artifact failure

Detection Engine

Witness-based candidate index: every rule is analyzed into witnesses, necessary conditions of which at least one holds on any event the rule can match, drawn from exact values, substring needles, keywords, mandatory regex literals, and field presence. The index inverts that relation over a per-field presence list and exact-value map, one Aho-Corasick automaton per field for substring needles, and one keyword automaton scanned against every string value. Rules with no sound witness, chiefly those whose only required branch is negated, are evaluated against every event. Candidates come back in ascending rule order.
Compiled matchers: optimized matching for all 30 modifier combinations — exact, contains, startswith, endswith, regex, CIDR, numeric comparison, base64 offset (3 alignment variants), windash expansion (5 replacement characters), field references, placeholder expansion, timestamp part extraction
Logsource-aware evaluation: opt-in LogSourceExtractor plus conflict-based pruning (set_logsource_extractor) skips rules whose product/service/category (and custom dimensions) conflict with the event's logsource, backed by a product-partitioned rule index; off by default and fail-open
Schema classification and routing: SchemaClassifier recognizes an event's schema from declarative field predicates (explain/classify_with_ambiguity for tuning), and SchemaRouter routes each event to its schema's pipeline engine, derives the event's logsource from the recognized schema for pruning, and feeds one shared correlation store
Condition tree evaluation: short-circuit boolean logic, selector patterns with quantifiers (1 of selection_*, all of them)
Filter application: runtime injection of filter rules as AND NOT conditions on referenced rules

Compilation Pipeline

Rule compilation (compile_rule): For each named detection, call compile_detection. Reads rsigma.include_event from custom_attributes.
Detection compilation (compile_detection):
- AllOf → compile each item, reject empty.
- AnyOf → recursively compile each sub-detection, reject empty.
- Keywords → compile each value as case-insensitive contains, combine with AnyOf. A number matches its decimal text.
- A field-less item such as '|all': [a, b] matches like a keyword: a value without a string operator is a substring, |all requires each value somewhere in the event, and |neq holds when no value matches. Inside an array body it matches the member itself. The keywords and conditions reference describes the semantics.
Value compilation (compile_value): |neq wraps the combined item in Not, so Field|neq: [a, b] matches when the field is neither a nor b, including when it is missing or null. The other modifiers are handled in this order: |expand → timestamp part → |fieldref → |re → |cidr → numeric comparison → string modifiers. Encoding modifiers transform the value in a fixed order: |windash variants first, then |wide/|utf16le, |utf16be, or |utf16, then |base64 or |base64offset, then the string match. |fieldref may be followed by one of |contains, |startswith, or |endswith; that modifier must come after |fieldref. An empty value list (Field: []) compiles to a null check. The value modifiers reference describes the matching semantics.

Compiled Matcher Types

Matcher	Modifier	Notes
`Exact`	(default)	Case-insensitive by default; `\|cased` makes it sensitive
`Contains`	`\|contains`	Substring match
`StartsWith`	`\|startswith`	Prefix match
`EndsWith`	`\|endswith`	Suffix match
`Regex`	`\|re`	`\|i` adds `(?i)`, `\|m` adds multiline, `\|s` adds dotall
`Cidr`	`\|cidr`	IP network matching via `IpNet`
`NumericEq/Gt/Gte/Lt/Lte`	`\|gt`, `\|gte`, etc.	f64 comparison
`Exists`	`\|exists`	Field presence; a field set to null is present. Accepts `true`/`yes`/`false`/`no` as values
`FieldRef`	`\|fieldref`	Compares against another field. Optional `\|contains`, `\|startswith`, or `\|endswith` after `\|fieldref`. Case-insensitive unless `\|cased`.
`Null`	—	Matches null or missing values
`BoolEq`	—	Boolean equality
`Expand`	`\|expand`	Placeholders left unresolved by the pipeline, filled from event fields at match time; honors `\|contains`, `\|startswith`, and `\|endswith`
`TimestampPart`	`\|minute`, `\|hour`, `\|day`, `\|week`, `\|month`, `\|year`	Extract timestamp component, match inner value
`Not`	`\|neq`	Wraps inner matcher with negation; matches a missing field unless the inner matcher does
`AnyOf` / `AllOf`	—	Multiple values combined (OR / AND with `\|all`)

Value Coercion

Arrays: string matchers use OR semantics (any element matches).
Numbers: coerced to string for string matchers.
Booleans: "true", "1", "yes" → true; "false", "0", "no" → false.

Filter Rule Behavior

Filters match filter.rules references by rule id or name. Matching by title remains as a deprecated compatibility fallback.
Selector patterns in filter conditions, including 1 of selection_* and all of them, are scoped to the filter's own detection items after injection. As in a detection rule, them and patterns that do not start with _ skip the filter's _-prefixed items.
If the filter has a logsource, each of category, product, and service that it specifies must match the target rule (case-insensitively); omitted fields act as wildcards, and custom logsource keys are not compared.
Empty filter.rules applies the filter to all rules.
Filter detections are added as __filter_{counter}_v_{name}, or __filter_{counter}_h_{name} for _-prefixed names (the counter prevents key collisions when multiple filters share detection names); the condition is wrapped as original AND filter condition, so suppression filters must write the negation explicitly.

Selector Pattern Matching

* — matches any detection name.
selection_* — prefix match.
*_filter — suffix match.
exact — exact match.
them — matches all names except those starting with _.
A pattern that does not start with _ also skips names starting with _.
A selector that matches no name is a compile error.

Event Model

The Event wrapper provides flexible field access over serde_json::Value:

Flat-key precedence: "actor.user.name" as a literal top-level key takes priority over nested traversal.
Dot-notation: if no flat key matches and the path contains ., split and traverse nested objects.
Array traversal: arrays are searched with OR semantics (first matching element wins).
Keyword detection: matches_keyword searches all string values across all fields recursively, and numbers as their JSON text.
Field enumeration: field_keys() returns the leaf field paths in dot-notation (e.g. actor.id), with intermediate object names included so callers can inspect coverage at any nesting level. Used by the daemon's opt-in field observer; not on the detection hot path. Ships with a default impl that walks to_json(); JsonEvent overrides with a zero-copy recursive walk.
Max nesting depth: recursive traversal stops at depth 64 (MAX_NESTING_DEPTH).

Correlation Engine

Stateful processing with sliding time windows, group-by aggregation, and all 8 correlation types.

CorrelationConfig

Field	Type	Default	Description
`timestamp_fields`	`Vec<String>`	`["@timestamp", "timestamp", "EventTime", "TimeCreated", "eventTime"]`	Field names to try for timestamp extraction, in priority order
`timestamp_fallback`	`TimestampFallback`	`WallClock`	`WallClock` (use `Utc::now()`) or `Skip` (skip event from correlation)
`max_state_entries`	`usize`	`100,000`	Hard cap across all correlations and group keys
`max_group_entries`	`Option<usize>`	`None`	Cap on retained entries within a single group's window state; `None` = unbounded. Oldest entries dropped on overflow (session windows keep their span anchor)
`suppress`	`Option<u64>`	`None`	Default suppression window in seconds
`action_on_match`	`CorrelationAction`	`Alert`	`Alert` (keep state) or `Reset` (clear window state)
`emit_detections`	`bool`	`false`	Whether to emit matches for correlation-only rules, including correlations referenced by another correlation
`correlation_event_mode`	`CorrelationEventMode`	`None`	`None`, `Full` (deflate-compressed), or `Refs` (timestamp + ID)
`max_correlation_events`	`usize`	`10`	Max events stored per `(correlation, group_key)` window

Core Features

Group-by partitioning: composite keys with field aliasing across referenced rules
Correlation chaining: correlation results propagate to higher-level correlations and those parent firings are emitted (max depth: 10, MAX_CHAIN_DEPTH). A correlation referenced by another correlation follows the generate flag below, so by default only the top of a chain is output
Temporal counting: temporal and temporal_ordered thresholds count distinct referenced rules, so a rule listed twice in rules counts once
Extended temporal conditions: boolean expressions over rule references (e.g. rule_a and rule_b and not rule_c)
Cycle detection: DFS-based validation of the correlation reference graph at load time

Alert Management

Suppression: per-correlation or global suppression windows to prevent alert floods. After a (correlation, group_key) fires, suppress re-alerts for the configured duration
Action-on-fire: Alert (keep state, re-fire on next match) or Reset (clear window state, require fresh threshold)
Generate flag: output of referenced detections and referenced correlations is suppressed by default; top-level generate: true on a correlation emits its referenced rules, and emit_detections: true emits every detection and correlation match

Event Inclusion

Full mode: contributing events stored as individually deflate-compressed blobs (compression level 1, 3-5x memory savings on typical JSON)
Refs mode: lightweight references (timestamp + optional event ID) at ~40 bytes per event
Event ID extraction (Refs mode): tries fields in order: id, _id, event_id, EventRecordID, event.id
Configurable cap: max_correlation_events bounds memory per window
Zero cost when disabled: buffers are not allocated unless mode is Full or Refs
Per-correlation override: set rsigma.correlation_event_mode via custom_attributes in YAML

Memory Management

Max state entries: configurable hard cap (default: 100,000) across all correlations and group keys
Time-based eviction: entries outside their correlation window are evicted automatically
Hard-cap eviction: when over the limit, entries are evicted until 90% of the cap is reached (the stalest 10% are dropped in bulk to avoid evicting on every event)
Stale alert cleanup: expired suppression entries are garbage-collected

Timestamp Extraction

Field priority list: configurable ordered list of fields to try (default: @timestamp, timestamp, EventTime, TimeCreated, eventTime)
Format support: RFC 3339, %Y-%m-%dT%H:%M:%S, %Y-%m-%dT%H:%M:%S%.f, %Y-%m-%d %H:%M:%S, epoch seconds, epoch milliseconds (auto-detected if value > 10^12)
Fallback policy: WallClock (use Utc::now(), good for real-time streaming) or Skip (skip event from correlation, recommended for batch/replay)

Value Percentile

value_percentile uses linear interpolation (C=1 method). The percentile field on the condition (0-100, clamped) selects which percentile to compute from the window's values; the predicate threshold (gte, lte, etc.) is then compared against that computed value. For example, percentile: 95 with lte: 100 fires when the 95th-percentile latency in the window is at most 100ms. An empty window has no percentile and the condition is not evaluated.

value_median is the 50th percentile under the same semantics. As with value_percentile, an empty window returns no value rather than 0.0, so predicates like lte: 0 cannot fire spuriously.

value_sum, value_avg, value_percentile, and value_median operate on a single numeric field. Declaring multiple fields under condition.field for these correlation types is rejected at compile time, since the Sigma specification does not define how to combine several numeric fields under one of these aggregations. value_count supports a list of fields and counts distinct tuples.

Output Types

EvaluationResult is the single output type produced by both detection and correlation. It composes a RuleHeader (shared rule metadata) and a ResultBody (kind-specific payload), both serialized into one flat JSON object via #[serde(flatten)]. Downstream consumers distinguish detection from correlation by presence of correlation_type (correlation-only) or matched_fields (detection-only). ProcessResult is an alias for Vec<EvaluationResult>.

EvaluationResult

Field	Type	Description
`header`	`RuleHeader`	Shared rule metadata, flattened on serialize
`body`	`ResultBody`	`Detection(DetectionBody)` or `Correlation(CorrelationBody)`, flattened on serialize

Accessors: is_detection() / is_correlation(), as_detection() -> Option<&DetectionBody>, as_correlation() -> Option<&CorrelationBody> (plus mutable variants).

RuleHeader

Field	Type	Description
`rule_title`	`String`	Rule title
`rule_id`	`Option<String>`	Rule UUID
`level`	`Option<Level>`	Severity level
`tags`	`Vec<String>`	Tags
`custom_attributes`	`Arc<HashMap<String, Value>>`	Custom attributes from the rule; skipped when empty
`enrichments`	`Option<Map<String, Value>>`	Optional enrichment map; skipped when `None`

DetectionBody

Field	Type	Description
`matched_selections`	`Vec<String>`	Detection names that matched
`matched_fields`	`Vec<FieldMatch>`	Field/value pairs that contributed to the match. Array object-scope matches record binding members with indexed paths (`connections[0].protocol`), capped at 32 members; `[none]` and vacuous `[all_or_empty]` keep the container.
`event`	`Option<Value>`	Full event JSON when `include_event` is enabled

CorrelationBody

Field	Type	Description
`correlation_type`	`CorrelationType`	e.g. `event_count`, `temporal`
`group_key`	`Vec<(String, String)>`	Group-by field/value pairs
`aggregated_value`	`f64`	Computed aggregate (count, sum, avg, percentile, median)
`timespan_secs`	`u64`	Correlation window duration
`events`	`Option<Vec<Value>>`	Contributing events (Full mode)
`event_refs`	`Option<Vec<EventRef>>`	Event references (Refs mode)

FieldMatch

Field	Type	Description
`field`	`String`	Field name that matched (`"keyword"` for keyword matches)
`value`	`serde_json::Value`	Event value that triggered the match (`null` for absence matches)
`selection`	`Option<String>`	Originating selection; populated above `MatchDetailLevel::Off`
`matcher`	`Option<MatcherKind>`	Matcher kind that fired (e.g. `contains`, `regex`, `one_of`)
`pattern`	`Option<String>`	Pattern the matcher tested against; `Full` level only, truncated
`case_sensitive`	`Option<bool>`	Whether matching was case-sensitive, when meaningful
`negated`	`bool`	`true` when the matcher was negated; omitted otherwise

The last five fields are populated by [Engine::set_match_detail]. The default MatchDetailLevel::Off records only {field, value} and skips the rest on serialization, so the wire shape is unchanged unless detail is enabled. Summary adds the descriptor fields and reports keyword and absence matches that Off omits; Full additionally records pattern.

EventRef

Field	Type
`timestamp`	`i64`
`id`	`Option<String>`

ProcessResultExt

Extension trait on [EvaluationResult] that exposes by-kind views without forcing pattern matching:

Method	Returns
`detections()`	`impl Iterator<Item = &EvaluationResult>` filtering to Detection-bodied results
`correlations()`	`impl Iterator<Item = &EvaluationResult>` filtering to Correlation-bodied results
`detection_count()`	`usize`
`correlation_count()`	`usize`

Bring rsigma_eval::ProcessResultExt into scope to use these methods.

Processing Pipelines

pySigma-compatible pipeline system for field mapping, logsource transformation, and backend-specific configuration. Supports multi-pipeline chaining with priority ordering.

Pipeline Chaining

Priority: Pipeline.priority (default 0); lower runs first.
Sorting: pipelines are sorted by priority on add.
State isolation: each pipeline gets its own PipelineState; state is not shared across pipelines. Use apply_pipelines_with_state() to collect merged state for backends.

Transformation Item Fields

Each transformation item in a pipeline can have:

Field	Description
`id`	Identifier for `processing_item_applied` conditions
`rule_conditions`	Rule conditions as a list or identifier-keyed mapping
`rule_cond_op` / `rule_cond_not` / `rule_cond_expr`	Link with `and`/`or`, negate the result, or evaluate an identifier expression; `rule_cond_expression` is an alias
`detection_item_conditions`	Conditions on individual detection items
`detection_item_cond_op` / `detection_item_cond_not` / `detection_item_cond_expr`	Detection-item condition linking, negation, and expressions
`field_name_conditions`	Conditions on field names
`field_name_cond_op` / `field_name_cond_not` / `field_name_cond_expr`	Field-name condition linking, negation, and expressions

List conditions have one-based identifiers (1, 2, ...). Condition expressions require every identifier to be defined and referenced. An expression and the corresponding _cond_op cannot be used together. Unknown transformation-item keys are rejected.

Condition semantics follow pySigma:

match_string and field-name regexes match from the start of the text, like Python's re.match. match_string sees a value as pySigma renders it after modifiers: contains, startswith, and endswith add their * wildcards and literal * and ? are escaped. Field references, regular expressions, CIDR values, numbers, comparisons, and encoded values never match; negate inverts each value's result before cond combines them.
Field-name conditions select a detection item when its field name or the target of one of its field references matches, and the negation applies to that combined result.
processing_item_applied at field-name scope is true for a fields entry or field reference target that the transformation with that id renamed, and the mark follows later renames. At detection-item scope it is true for a detection item that the transformation changed, and an item keeps the marks of the item it replaced.
processing_state eq and ne compare numbers by value, so 1 equals 1.0.

Transformations (26 types)

Type	Fields	Description
`field_name_mapping`	`mapping: {k: v \| [v1, v2, ...]}`	Rename fields, `fieldref` targets, and the rule's `fields` list via a mapping dict; list values expand the matched detection item into an OR over the alternatives (one-to-many, pySigma-compatible)
`field_name_prefix_mapping`	`mapping: {prefix: replacement}`	Rename fields matching a prefix
`field_name_prefix`	`prefix`	Add a prefix to all field names
`field_name_suffix`	`suffix`	Add a suffix to all field names
`field_name_transform`	`transform_func`, `mapping`, `apply_keyword` (only `false`)	Case transformation (see below)
`drop_detection_item`	—	Remove matching detection items
`add_condition`	`conditions: {k: v \| [v1, v2, ...]}`, `negated` (default: `false`), `name`, `template` (default: `false`)	Inject additional detection conditions; list values are OR-linked. `name` names the added detection and must not collide with an existing one; `template: true` substitutes `$category`, `$product`, and `$service` in string values
`change_logsource`	`category`, `product`, `service`	Modify logsource fields
`replace_string`	`regex`, `replacement`, `skip_special` (default: `false`), `interpret_special` (default: `false`)	Regex-based string replacement (`skip_special` preserves wildcards; `interpret_special` parses wildcards in the replaced text)
`map_string`	`mapping: {k: v \| [v1, v2]}`	Map string values to replacements (supports one-to-many)
`set_value`	`value`, `force_type` (`str`/`num`)	Replace detection item values
`convert_type`	`target_type` (`str`/`int`/`float`/`bool`, default: `str`)	Convert values between types
`value_placeholders`	`allow_unresolved` (default: `false`), `include`/`exclude`	Expand `%placeholder%` in `
`wildcard_placeholders`	`include`/`exclude`	Replace handled placeholders with wildcards in `
`query_expression_placeholders`	`expression` (default: `""`)	Backend query placeholders (no-op in eval); `mapping`, `include`, and `exclude` are rejected
`set_state`	`key`, `val` (`value` alias)	Store typed key-value pairs in pipeline state
`rule_failure`	`message` (default: `"rule failure"`)	Raise an error for matching rules
`detection_item_failure`	`message` (default: `"detection item failure"`)	Raise an error for matching detection items
`hashes_fields`	`valid_hash_algos` (required), `field_prefix` (default: `""`), `drop_algo_prefix` (default: `false`), `field_to_parse` (default: `[Hashes, Hash]`)	Replace hash items with an OR over per-algorithm fields; values are `ALGO=hash`, `ALGO\|hash`, or a bare hash whose algorithm comes from its length
`add_field`	`field`	Add a new detection item with a fixed value
`remove_field`	`field`	Remove a field from detection items
`set_field`	`fields: [...]`	Rename the field of a detection item
`set_custom_attribute`	`attribute`, `value`	Set key-value attributes on rules
`case_transformation`	`case_type` / `case` (`lower`/`upper`/`snake_case`)	Transform case of field values
`nest`	`items` or `transformations`	Apply a group of transformations conditionally
`regex`	`method` (`ignore_case_brackets`/`ignore_case_flag`; `plain` is rejected)	Regex transformation (no-op in eval, where string matching is already case-insensitive)

Aliases: case is accepted as an alias for case_transformation.

`field_name_transform` Functions

Value	Behavior
`lower` / `lowercase`	`to_lowercase`
`upper` / `uppercase`	`to_uppercase`
`title`	Capitalize each word, join with `_` (e.g. `hello_world` → `Hello_World`)
`snake_case`	camelCase → snake_case

Conditions (3 levels)

Rule Conditions

Type	Fields
`logsource`	`category`, `product`, `service`
`contains_detection_item`	`field`, `value` (optional)
`processing_item_applied`	`processing_item_id`
`processing_state`	`key`, `val`, `op` (`eq`/`ne`/`gte`/`gt`/`lte`/`lt`)
`is_sigma_rule`	—
`is_sigma_correlation_rule`	—
`rule_attribute`	`attribute` (`level`/`status`/`author`/`title`/`id`/`date`/`description`), `value`
`tag`	`tag`

Detection Item Conditions

Type	Fields
`match_string`	`pattern` (default: `".*"`), `negate` (default: `false`), `cond` (`any`/`all`, default: `any`)
`is_null`	`negate`, `cond` (`any`/`all`, default: `any`)
`processing_item_applied`	`processing_item_id`
`processing_state`	`key`, `val`, `op` (`eq`/`ne`/`gte`/`gt`/`lte`/`lt`)

Field Name Conditions

Type	Fields
`include_fields`	`fields`, `mode` (`plain` or `re`, default: `plain`; `match_type` is an alias)
`exclude_fields`	`fields`, `mode`
`processing_item_applied`	`processing_item_id`
`processing_state`	`key`, `val`, `op` (`eq`/`ne`/`gte`/`gt`/`lte`/`lt`)

Finalizers (3 types)

Type	Fields	Defaults
`concat`	`separator`, `prefix`, `suffix`	`" "`, `""`, `""`
`json`	`indent`	—
`template`	`template`	`""`

Finalizers are stored in the pipeline and not executed in eval mode. Each finalizer has an apply() method used by rsigma-convert backends to transform a Vec<String> of queries into a single output string.

Custom Attributes (`rsigma.*`)

Pipeline transformations can configure engine behavior via SetCustomAttribute, following the same pattern as pySigma backends (e.g. pySigma-backend-loki):

Attribute	Effect	CLI equivalent	Scope
`rsigma.timestamp_field`	Prepends a field name to the timestamp extraction priority list	`--timestamp-field`	Engine
`rsigma.suppress`	Sets the suppression window (e.g. `5m`)	`--suppress`	Engine + per-correlation
`rsigma.action`	Sets the post-fire action (`alert` or `reset`)	`--action`	Engine + per-correlation
`rsigma.include_event`	Embeds the full event JSON in detection output	`--include-event`	Per-rule
`rsigma.correlation_event_mode`	Sets event inclusion mode (`full` or `refs`)	`--correlation-event-mode`	Per-correlation
`rsigma.max_correlation_events`	Caps stored events per correlation window (integer)	`--max-correlation-events`	Per-correlation
`rsigma.max_group_entries`	Caps retained entries within a single group's window state (integer, quoted)	`--max-group-entries`	Per-correlation

CLI flags and the library API always take precedence over pipeline attributes. Engine-level attributes (timestamp_field, suppress, action) are only applied when the CLI did not already set the corresponding flag. Per-correlation attributes override engine defaults for individual correlation rules.

# Example pipeline with custom attributes
transformations:
  - type: set_custom_attribute
    attribute: rsigma.timestamp_field
    value: time
  - type: set_custom_attribute
    attribute: rsigma.suppress
    value: 5m

Bloom Pre-Filter (Opt-In)

The engine can build a per-field bloom filter at rule-load time over every positive substring needle (Contains / StartsWith / EndsWith / AhoCorasickSet). When enabled, Engine::evaluate short-circuits any positive substring detection item whose field value cannot possibly contain a needle trigram, skipping the matcher entirely.

Off by default. The per-event probe (trigram extraction + double hashing) costs ~1 µs on a typical CommandLine field. On rule sets where most events overlap with at least one needle, the probe is pure overhead. The bloom only pays off on substring-heavy rule sets paired with mostly-non-matching events (e.g. high-volume telemetry against an active threat-intel ruleset).

let mut engine = Engine::new();
engine.set_bloom_prefilter(true);
// Optional: tighten or relax the 1 MB default budget.
engine.set_bloom_max_bytes(2 * 1024 * 1024);
engine.add_collection(&collection)?;

CLI equivalents on rsigma engine eval and rsigma engine daemon:

--bloom-prefilter              # enable
--bloom-max-bytes <BYTES>      # override 1 MB default

Always benchmark against representative events before flipping it on; the eval_bloom_rejection Criterion group in crates/rsigma-eval/benches/eval.rs reports throughput with both default-off and bloom-on engines so you can size the win on your corpus.

Cross-Rule Aho-Corasick Index (Opt-In, Feature-Gated)

For deployments with very large rule sets (> ~5K rules) and many shared substring patterns (threat-intel feeds, IOC packs), the engine can build a single per-field DoubleArrayAhoCorasick automaton over every rule's positive substring needles. At eval time, the engine scans each indexed field once with the per-field automaton and drops AC-prunable rules from the candidate set when none of their patterns hit the event.

A rule is AC-prunable when:

It has at least one positive substring detection item, AND
Every detection consists exclusively of positive substring matchers (Contains / StartsWith / EndsWith / AhoCorasickSet, possibly nested under AnyOf/AllOf/CaseInsensitiveGroup), AND
No condition expression contains Not.

These rules can be safely pruned because their firing requires at least one substring to match. Rules with Exact, Regex, Numeric, Cidr, etc. matchers, or with not selectors in their conditions, are kept in the candidate set unfiltered.

Off by default. For smaller rule sets the per-rule [AhoCorasickSet] matcher is already optimal; the cross-rule index only adds build time and lookup overhead. Pattern count per field is capped at 100K (rules referencing fields above that cap are kept unfiltered). Build time scales linearly with total pattern count.

Enable via the daachorse-index Cargo feature:

rsigma-eval = { version = "0.10", features = ["daachorse-index"] }

let mut engine = Engine::new();
engine.set_cross_rule_ac(true);
engine.add_collection(&collection)?;

CLI equivalents on rsigma engine eval and rsigma engine daemon (when the CLI is compiled with the feature):

--cross-rule-ac

Always benchmark against representative rule sets and event streams before flipping it on; the eval_cross_rule_ac Criterion group reports throughput at 1K / 5K / 10K rules with the index on and off.

Constants and Limits

Constant	Value	Purpose
`MAX_NESTING_DEPTH`	64	Recursive JSON traversal depth for keyword search
`MAX_WINDASH_DASHES`	8	Maximum dash characters expanded by windash (5^8 variants); defined in `rsigma_ir::encoding` and shared with conversion
Windash characters	5	`-`, `/`, `–` (en-dash), `—` (em-dash), `―` (horizontal bar)
`MAX_CHAIN_DEPTH`	10	Maximum correlation chaining depth
`max_state_entries`	100,000	Default hard cap for correlation state
Eviction target	90%	Hard-cap eviction drops the stalest 10%
`max_correlation_events`	10	Default per-window event cap
Epoch threshold	10^12	Numeric timestamps above this are treated as milliseconds

Error Types

Error	When
`InvalidRegex`	Regex compilation failure
`InvalidCidr`	CIDR parse failure in IR that did not come from lowering a rule
`IncompatibleValue`	A value the modifier cannot use, such as a `cidr` with host bits set or a wildcard in a field reference
`Base64`	Base64 encoding error
`UnknownDetection`	Condition references missing detection (caught at compile time)
`InvalidModifiers`	Invalid modifier combo, empty AllOf/AnyOf, windash overflow, pipeline failure
`IncompatibleValue`	Wrong type for modifier (e.g. null for string)
`ExpectedNumeric`	Numeric modifier with non-numeric value
`Parser`	Parser error (from rsigma-parser)
`CorrelationError`	Correlation compile/runtime error
`UnknownRuleRef`	Correlation references unknown rule (caught at `add_collection` time)
`CorrelationCycle`	Cycle in correlation references

Usage

Detection only:

use rsigma_parser::parse_sigma_yaml;
use rsigma_eval::{Engine, parse_pipeline};
use rsigma_eval::event::JsonEvent;
use serde_json::json;

let yaml = r#"
title: Detect Whoami
logsource:
    product: windows
    category: process_creation
detection:
    selection:
        CommandLine|contains: 'whoami'
    condition: selection
level: medium
"#;

let collection = parse_sigma_yaml(yaml).unwrap();

let pipeline = parse_pipeline(r#"
name: ECS Mapping
transformations:
  - type: field_name_mapping
    mapping:
      CommandLine: process.command_line
    rule_conditions:
      - type: logsource
        product: windows
"#).unwrap();

let mut engine = Engine::new_with_pipeline(pipeline);
engine.add_collection(&collection).unwrap();

// Rule now expects ECS field names
let event = JsonEvent::borrow(&json!({"process.command_line": "whoami"}));
let matches = engine.evaluate(&event);

field_name_mapping also accepts a list of alternatives, matching pySigma's FieldMappingTransformation. The matched detection item is expanded into an OR over the alternatives — when the surrounding AllOf selection has other items, they're preserved across each branch via a Cartesian expansion so the AND / OR semantics stay correct:

name: Hashes mapping
transformations:
  - type: field_name_mapping
    mapping:
      Hashes:
        - file.hash.md5
        - file.hash.sha1
        - file.hash.sha256

After applying this pipeline, a rule selecting Hashes: 'abc123' matches an event populating any of file.hash.md5, file.hash.sha1, or file.hash.sha256. Correlation rules (group_by, aliases, threshold field) consume only the first listed alternative since those positions are inherently scalar.

With correlations:

use rsigma_eval::{CorrelationEngine, CorrelationConfig, CorrelationAction, CorrelationEventMode};

let config = CorrelationConfig {
    suppress: Some(300),                         // 5-minute suppression window
    action_on_match: CorrelationAction::Reset,   // clear state after firing
    emit_detections: false,                      // only emit correlation alerts
    correlation_event_mode: CorrelationEventMode::Full, // include full events
    max_correlation_events: 20,                        // keep last 20 events per window
    ..Default::default()
};

let mut engine = CorrelationEngine::new(config);
engine.set_include_event(true);                  // embed event JSON in all match results
engine.add_collection(&collection).unwrap();
let result = engine.process_event_at(&event, timestamp_secs);
// result: Vec<EvaluationResult>, detections-then-correlations in evaluation order
// result.detections() / result.correlations(): filtered iterators (ProcessResultExt)
// result.correlations().next().unwrap().as_correlation().unwrap().events:     (Full mode)
// result.correlations().next().unwrap().as_correlation().unwrap().event_refs: (Refs mode)

Benchmarks

Criterion.rs benchmarks with synthetic rules and events (Apple M-series, single-threaded unless noted). Rules are pre-filtered at evaluation time by the witness-based candidate index. The tables below were captured against the earlier exact-value-only index, so the Indexed column understates current throughput on the wildcard-heavy and regex-heavy scenarios, which the witness index prunes and its predecessor did not. For measured numbers on a real corpus see the SigmaHQ corpus baseline, which records single-core and daemon throughput before and after witness indexing on four event shapes.

Detection Evaluation

Scenario	Baseline	Indexed	Speedup
Compile 1,000 rules	740 µs	946 µs	0.8x (index build)
Compile 5,000 rules	3.8 ms	4.7 ms	0.8x (index build)
1 event vs 100 rules	5.6 µs	2.3 µs	2.4x
1 event vs 1,000 rules	81 µs	30 µs	2.7x
1 event vs 5,000 rules	415 µs	164 µs	2.5x
100K events vs 100 rules	613 ms (163K/s)	253 ms (396K/s)	2.4x
Wildcard-heavy (1,000 rules, 100 events)	16 µs	18 µs	~1x (unindexable)
Regex-heavy (1,000 rules, 100 events)	3.3 µs	5.1 µs	~1x (unindexable)

Batch Evaluation (sequential vs `parallel` feature)

Rules	Events	Sequential	Batch
100	1,000	2.5 ms	2.5 ms
1,000	1,000	30.5 ms	31.0 ms
5,000	1,000	162 ms	164 ms

Correlation Engine

Scenario	Baseline	Indexed	Speedup
1K events, 20 event_count correlations	1.08 ms (926K/s)	988 µs (1.01M/s)	1.1x
1K events, 10 temporal correlations	489 µs (2.04M/s)	502 µs (1.99M/s)	~1x
100K events, 50 detection + 10 correlation rules	293 ms (342K/s)	176 ms (568K/s)	1.7x
Batch 10K events (sequential)	--	18.1 ms (552K/s)	--
Batch 10K events (process_batch)	--	19.2 ms (521K/s)	--
50K unique group keys (state pressure)	32.9 ms (1.52M/s)	38.4 ms (1.30M/s)	~1x

cargo bench --bench eval
cargo bench --bench correlation

License

MIT License.

rsigma-eval 0.24.0

rsigma-eval

Public API

Engine

Correlation Engine

Compilation

Explain (explain module)

Pipeline

Rule field extraction (fields module)

Field observability (field_observer module)

Rule drafting (rule_draft module)

Rule exemplars (exemplar module)

Rule tuning (rule_tune module)

Detection Engine

Compilation Pipeline

Compiled Matcher Types

Value Coercion

Filter Rule Behavior

Selector Pattern Matching

Event Model

Correlation Engine

CorrelationConfig

Core Features

Alert Management

Event Inclusion

Memory Management

Timestamp Extraction

Value Percentile

Output Types

EvaluationResult

RuleHeader

DetectionBody

CorrelationBody

FieldMatch

EventRef

ProcessResultExt

Processing Pipelines

Pipeline Chaining

Transformation Item Fields

Transformations (26 types)

field_name_transform Functions

Conditions (3 levels)

Rule Conditions

Detection Item Conditions

Field Name Conditions

Finalizers (3 types)

Custom Attributes (rsigma.*)

Bloom Pre-Filter (Opt-In)

Cross-Rule Aho-Corasick Index (Opt-In, Feature-Gated)

Constants and Limits

Error Types

Usage

Benchmarks

Detection Evaluation

Batch Evaluation (sequential vs parallel feature)

Correlation Engine

License

Explain (`explain` module)

Rule field extraction (`fields` module)

Field observability (`field_observer` module)

Rule drafting (`rule_draft` module)

Rule exemplars (`exemplar` module)

Rule tuning (`rule_tune` module)

`field_name_transform` Functions

Custom Attributes (`rsigma.*`)

Batch Evaluation (sequential vs `parallel` feature)