Crate iqdb_filter 

iqdb-filter

Canonical iqdb_types::Filter evaluator for the HiveDB iqdb vector-database spine. One place that decides what Filter means; every index that supports metadata filtering delegates to it.

Why this lives outside the index crates

Filtering used to be inlined in iqdb-flat. The moment a second index (HNSW, IVF) starts honouring filters, two copies of the semantics would drift — the Neq(absent) / Not(Eq(absent)) rule is exactly the kind of subtlety that splits between implementations and produces query-result bugs nobody can attribute. Extracting the evaluator pins one set of semantics across every consumer.

Public surface

• FilterEvaluator — new(filter) -> Result<Self, IqdbError> validates the filter once (depth, In cardinality); evaluate(metadata) -> bool is infallible on a validated filter. FilterEvaluator::prefilter and FilterEvaluator::postfilter apply it as lazy, allocation-free scan adapters over a stream of (key, metadata) pairs.
• MetadataIndex — an opt-in, per-field inverted index that resolves a selective Eq / In predicate to a candidate key set (a superset of the true matches), and backs a sharper, count-based selectivity estimate.
• estimate_selectivity — a best-effort, structural estimate of the fraction of records a validated filter passes, in [0.0, 1.0]; the index-backed counterpart is MetadataIndex::estimate_selectivity.
• choose_strategy / StrategySelector — pick a concrete FilterStrategy from the selectivity estimate. The free function uses the DEFAULT_PREFILTER_THRESHOLD; the selector is the Tier-2 builder for tuning it.
• FilterStrategy — vocabulary for how an index applies a filter relative to its distance scan. The selector resolves Auto down to PreFilter / PostFilter; InFilter waits on a graph-index consumer.
• MAX_FILTER_DEPTH / MAX_IN_VALUES — documented validation caps, pub const so callers can quote them in error messages or higher-level validation.

Null and absent-field semantics

The evaluator implements the closed-world rule pinned by iqdb_types::Filter: every leaf comparison (Eq, Neq, Lt, Lte, Gt, Gte, In) over a field absent from the record’s metadata evaluates to false. Type mismatches between a stored value and a literal also evaluate to false. Value::Float(NaN) under any ordered comparison evaluates to false (IEEE-754 unordered). Not over a false leaf is true, which is the idiom for “records without this field, or with a non-matching value.”

Neq(absent) → false and Not(Eq(absent)) → true are therefore not interchangeable. The pair is pinned by the conformance tests in tests/conformance.rs.

DoS hardening

Construction is the validation gate. The walk is iterative (an explicit stack, not recursion), so new cannot itself stack-overflow on adversarial input. After construction every filter is bounded by MAX_FILTER_DEPTH, so the recursive FilterEvaluator::evaluate hot path runs with a bounded call stack.

Example

use iqdb_filter::FilterEvaluator;
use iqdb_types::{Filter, Metadata, Value};

let filter = Filter::and(vec![
    Filter::eq("published", Value::Bool(true)),
    Filter::gt("year", Value::Int(2000)),
]);
let evaluator = FilterEvaluator::new(filter)?;

let meta: Metadata = [
    ("published".to_string(), Value::Bool(true)),
    ("year".to_string(), Value::Int(2026)),
]
.into_iter()
.collect();

assert!(evaluator.evaluate(Some(&meta)));
assert!(!evaluator.evaluate(None));

Structs

FilterEvaluator

A validated Filter paired with the canonical evaluator.

MetadataIndex

An opt-in, per-field inverted index over record metadata.

StrategySelector

Picks a concrete FilterStrategy for a validated filter from its estimated selectivity — the Tier-2, tunable counterpart to choose_strategy.

Enums

FilterStrategy

How an index plans to apply a metadata Filter relative to its distance scan.

Constants

DEFAULT_PREFILTER_THRESHOLD

Default cutoff StrategySelector uses to split PreFilter from PostFilter: a filter whose estimated selectivity is at or below this value is treated as narrow enough to pre-filter.

MAX_FILTER_DEPTH

Maximum allowed nesting depth of a Filter passed to FilterEvaluator::new.

MAX_IN_VALUES

Maximum allowed number of values in a single Filter::In node.

VERSION

The version of this crate, taken from Cargo.toml at compile time.

Functions

choose_strategy

Picks a concrete FilterStrategy for a validated filter using the DEFAULT_PREFILTER_THRESHOLD — the Tier-1 shortcut for StrategySelector::new().choose(..).

estimate_selectivity

Estimate the fraction of records evaluator’s filter will pass, in [0.0, 1.0] — 0.0 means “matches almost nothing”, 1.0 means “matches (almost) everything”.