policy-rs

Rust implementation of the Tero Policy Specification for high-performance log policy evaluation and transformation.

Another implementation of this specification is available in Tero Edge, a Zig-based observability edge runtime, providing the policy evaluation engine for filtering, sampling, and transforming telemetry data.

Features

• High-performance pattern matching using Hyperscan for parallel regex evaluation
• Policy-based log filtering with keep, drop, sample, and rate-limit actions
• Log transformations including field removal, redaction, renaming, and addition
• Multiple policy providers with live reload support
• Zero-allocation field access through the Matchable trait
• Async-first design built on Tokio

Installation

Add to your Cargo.toml:

[dependencies]
policy-rs = { git = "https://github.com/usetero/policy-rs" }

Quick Start

use policy_rs::{EvaluateResult, FileProvider, PolicyEngine, PolicyRegistry, Matchable, LogFieldSelector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create registry and load policies
    let registry = PolicyRegistry::new();
    let provider = FileProvider::new("policies.json");
    registry.subscribe(&provider)?;

    // Create engine and get snapshot
    let engine = PolicyEngine::new();
    let snapshot = registry.snapshot();

    // Evaluate a log record
    let log = MyLogRecord::new("Error: connection timeout", "ERROR");
    let result = engine.evaluate(&snapshot, &log).await?;

    match result {
        EvaluateResult::NoMatch => println!("Pass through"),
        EvaluateResult::Keep { policy_id, .. } => println!("Keep: {}", policy_id),
        EvaluateResult::Drop { policy_id } => println!("Drop: {}", policy_id),
        EvaluateResult::Sample { keep, .. } => println!("Sampled: {}", keep),
        EvaluateResult::RateLimit { allowed, .. } => println!("Rate limited: {}", allowed),
    }

    Ok(())
}

Core Concepts

Policy Registry

The PolicyRegistry manages policies from multiple providers and maintains an immutable snapshot for lock-free evaluation:

let registry = PolicyRegistry::new();

// Subscribe to a file-based provider (auto-reloads on changes)
let provider = FileProvider::new("policies.json");
registry.subscribe(&provider)?;

// Or register a custom provider
let handle = registry.register_provider();
handle.update(vec![policy1, policy2]);

// Get immutable snapshot for evaluation
let snapshot = registry.snapshot();

Policy Engine

The PolicyEngine evaluates logs against compiled policies using Hyperscan for pattern matching:

let engine = PolicyEngine::new();
let snapshot = registry.snapshot();

// Read-only evaluation
let result = engine.evaluate(&snapshot, &log).await?;

// Evaluation with transformations applied
let result = engine.evaluate_and_transform(&snapshot, &mut log).await?;

Evaluation Results

pub enum EvaluateResult {
    /// No policies matched - pass through unchanged
    NoMatch,
    /// Matched policy says keep
    Keep { policy_id: String, transformed: bool },
    /// Matched policy says drop
    Drop { policy_id: String },
    /// Matched policy says sample (percentage-based)
    Sample { policy_id: String, percentage: f64, keep: bool, transformed: bool },
    /// Matched policy says rate limit (count-based)
    RateLimit { policy_id: String, allowed: bool, transformed: bool },
}

Implementing the Traits

To evaluate your log types, implement the Matchable trait. For transformation support, also implement Transformable.

Matchable Trait

The Matchable trait provides field access for pattern matching with two primitives: get_field returns the field's string value (for regex / equals / contains matchers), and field_exists reports presence regardless of value type (for exists: true matchers).

use std::borrow::Cow;
use policy_rs::{LogFieldSelector, LogSignal, Matchable};
use policy_rs::proto::tero::policy::v1::LogField;

struct MyLogRecord {
    body: String,
    severity: String,
    attributes: HashMap<String, String>,
}

impl Matchable for MyLogRecord {
    type Signal = LogSignal;

    fn get_field(&self, field: &LogFieldSelector) -> Option<Cow<'_, str>> {
        match field {
            LogFieldSelector::Simple(LogField::Body) => Some(Cow::Borrowed(&self.body)),
            LogFieldSelector::Simple(LogField::SeverityText) => Some(Cow::Borrowed(&self.severity)),
            LogFieldSelector::LogAttribute(path) => path
                .first()
                .and_then(|key| self.attributes.get(key))
                .map(|s| Cow::Borrowed(s.as_str())),
            _ => None,
        }
    }
}

The default field_exists is self.get_field(field).is_some(), which is correct as long as every present value is a string. If your records carry non-string values (numbers, booleans, structured values), override field_exists so exists: true matchers fire on those attributes — a record whose count: 42 lives only as an integer would otherwise be reported as absent because get_field cannot return a string for it.

For example, a record that holds OTel-style typed values:

enum AnyValue {
    String(String),
    Int(i64),
    Bool(bool),
}

struct OtelLogRecord {
    body: String,
    attributes: HashMap<String, AnyValue>,
}

impl Matchable for OtelLogRecord {
    type Signal = LogSignal;

    fn get_field(&self, field: &LogFieldSelector) -> Option<Cow<'_, str>> {
        match field {
            LogFieldSelector::Simple(LogField::Body) => Some(Cow::Borrowed(&self.body)),
            LogFieldSelector::LogAttribute(path) => match path
                .first()
                .and_then(|key| self.attributes.get(key))?
            {
                AnyValue::String(s) => Some(Cow::Borrowed(s.as_str())),
                // Int/Bool aren't representable as a borrowed &str — return
                // None and rely on field_exists for presence checks.
                _ => None,
            },
            _ => None,
        }
    }

    fn field_exists(&self, field: &LogFieldSelector) -> bool {
        match field {
            LogFieldSelector::LogAttribute(path) => path
                .first()
                .map(|key| self.attributes.contains_key(key))
                .unwrap_or(false),
            _ => self.get_field(field).is_some(),
        }
    }
}

Without the override, a policy with exists: true on count (stored as AnyValue::Int(42)) would not fire, because get_field correctly returns None for a value that can't be expressed as a string.

Transformable Trait

The Transformable trait exposes three minimal write primitives — set_field, delete_field, and move_field. The engine composes these with the read side of Matchable to drive higher-level transform ops (regex redact, upsert add, rename-with-upsert), so consumers don't have to express upsert checks or regex matching themselves.

use policy_rs::{LogFieldSelector, Transformable};
use policy_rs::proto::tero::policy::v1::LogField;

impl Transformable for MyLogRecord {
    fn set_field(&mut self, field: &LogFieldSelector, value: &str) {
        match field {
            LogFieldSelector::Simple(LogField::Body) => {
                self.body = value.to_string();
            }
            LogFieldSelector::Simple(LogField::SeverityText) => {
                self.severity = value.to_string();
            }
            LogFieldSelector::LogAttribute(path) => {
                if let Some(key) = path.first() {
                    self.attributes.insert(key.clone(), value.to_string());
                }
            }
            _ => {}
        }
    }

    fn delete_field(&mut self, field: &LogFieldSelector) -> bool {
        match field {
            LogFieldSelector::LogAttribute(path) => path
                .first()
                .and_then(|key| self.attributes.remove(key))
                .is_some(),
            _ => false,
        }
    }

    fn move_field(&mut self, from: &LogFieldSelector, to: &LogFieldSelector) {
        let value = match from {
            LogFieldSelector::LogAttribute(path) => {
                path.first().and_then(|key| self.attributes.remove(key))
            }
            _ => None,
        };
        let Some(v) = value else { return };
        if let LogFieldSelector::LogAttribute(path) = to
            && let Some(key) = path.first()
        {
            self.attributes.insert(key.clone(), v);
        }
    }
}

The engine constructs to so its variant matches the source's attribute namespace — e.g. renaming a ResourceAttribute produces a target selector of ResourceAttribute. Implementors should dispatch on to's variant rather than assuming a primary namespace.

Advanced Usage

Custom Policy Providers

Implement PolicyProvider to load policies from custom sources:

use policy_rs::{PolicyProvider, PolicyCallback, Policy, PolicyError};

struct MyProvider {
    // Your state here
}

impl PolicyProvider for MyProvider {
    fn load(&self, callback: &PolicyCallback) -> Result<(), PolicyError> {
        let policies = self.fetch_policies()?;
        callback.update(policies);
        Ok(())
    }
}

// Use with the registry
let registry = PolicyRegistry::new();
let provider = MyProvider::new();
registry.subscribe(&provider)?;

Policy Statistics

Track policy hit/miss rates and transform statistics:

let snapshot = registry.snapshot();

for entry in snapshot.iter() {
    let stats = entry.stats.snapshot();

    println!("Policy: {}", entry.policy.id());
    println!("  Matches: {} hits, {} misses", stats.match_hits, stats.match_misses);
    println!("  Remove: {} hits, {} misses", stats.remove.0, stats.remove.1);
    println!("  Redact: {} hits, {} misses", stats.redact.0, stats.redact.1);
    println!("  Rename: {} hits, {} misses", stats.rename.0, stats.rename.1);
    println!("  Add: {} hits, {} misses", stats.add.0, stats.add.1);
}

Multiple Providers

Combine policies from multiple sources:

let registry = PolicyRegistry::new();

// File-based policies
let file_provider = FileProvider::new("local-policies.json");
registry.subscribe(&file_provider)?;

// Programmatic policies
let handle = registry.register_provider();
handle.update(vec![
    create_emergency_drop_policy(),
    create_rate_limit_policy(),
]);

// All policies are merged in the snapshot
let snapshot = registry.snapshot();

Configuration-Based Providers

Use the config module to define providers in JSON/TOML configuration files. The ProviderConfig type is designed to be embedded in your application's config:

use policy_rs::config::{ProviderConfig, register_providers};
use policy_rs::PolicyRegistry;
use serde::Deserialize;

#[derive(Deserialize)]
struct AppConfig {
    service_name: String,
    policy_providers: Vec<ProviderConfig>,
}

// Parse your app config
let config: AppConfig = serde_json::from_str(r#"{
    "service_name": "my-app",
    "policy_providers": [
        {
            "id": "local",
            "type": "file",
            "path": "policies.json"
        },
        {
            "id": "remote",
            "type": "http",
            "url": "https://api.example.com/policies",
            "headers": [
                { "name": "Authorization", "value": "Bearer token123" }
            ],
            "poll_interval_secs": 60
        }
    ]
}"#)?;

// Register all providers at once
let registry = PolicyRegistry::new();
register_providers(&config.policy_providers, &registry)?;

Provider Config Format

Each provider configuration has a type field that determines the provider:

File Provider:

{
  "id": "local-policies",
  "type": "file",
  "path": "policies.json"
}

HTTP Provider (requires http feature):

{
  "id": "remote-policies",
  "type": "http",
  "url": "https://api.example.com/policies",
  "headers": [{ "name": "Authorization", "value": "Bearer token" }],
  "poll_interval_secs": 60,
  "content_type": "application/json"
}

gRPC Provider (requires grpc feature):

{
  "id": "grpc-policies",
  "type": "grpc",
  "endpoint": "https://grpc.example.com:443"
}

You can also parse just the provider list directly:

let providers: Vec<ProviderConfig> = serde_json::from_str(r#"[
    { "id": "file", "type": "file", "path": "policies.json" }
]"#)?;

Transform Order

When using evaluate_and_transform, transformations are applied in a fixed order:

1. Remove - Delete fields
2. Redact - Replace field values with placeholders
3. Rename - Rename fields to new keys
4. Add - Add new fields

Transforms from all matching policies are applied, not just the winning policy.

Policy Format

Policies are defined using the Tero Policy protobuf schema. Example JSON:

{
  "id": "drop-debug-logs",
  "name": "Drop Debug Logs",
  "enabled": true,
  "target": {
    "log": {
      "match": [
        {
          "logField": "SEVERITY_TEXT",
          "regex": "DEBUG|TRACE"
        }
      ],
      "keep": "none"
    }
  }
}

Keep Values

• "all" - Keep all matching logs
• "none" - Drop all matching logs
• "50%" - Sample 50% of matching logs
• "100/s" - Rate limit to 100 logs per second
• "1000/m" - Rate limit to 1000 logs per minute

Match Fields

• logField - Simple fields: BODY, SEVERITY_TEXT, TRACE_ID, SPAN_ID, etc.
• logAttribute - Log attributes by key
• resourceAttribute - Resource attributes by key
• scopeAttribute - Scope attributes by key

Match Types

• exact - Exact string match
• regex - Regular expression match
• exists - Field existence check

Examples

See the examples/ directory:

• basic_usage.rs - Load policies and evaluate logs
• transforms.rs - Apply log transformations
• multiple_providers.rs - Combine multiple policy sources
• custom_provider.rs - Implement a custom provider
• config_providers.rs - Configure providers via JSON config

Run examples with:

cargo run --example basic_usage
cargo run --example transforms
cargo run --example config_providers

License

Apache-2.0