policy-rs

Rust implementation of the Tero Policy Specification for high-performance log policy evaluation and transformation.

Another implementation of this specification is available in Tero Edge, a Zig-based observability edge runtime, providing the policy evaluation engine for filtering, sampling, and transforming telemetry data.

Features

• High-performance pattern matching using Hyperscan for parallel regex evaluation
• Policy-based log filtering with keep, drop, sample, and rate-limit actions
• Log transformations including field removal, redaction, renaming, and addition
• Multiple policy providers with live reload support
• Zero-allocation field access through the Matchable trait
• Async-first design built on Tokio

Installation

Add to your Cargo.toml:

[dependencies]
policy-rs = { git = "https://github.com/usetero/policy-rs" }

Quick Start

use policy_rs::{EvaluateResult, FileProvider, PolicyEngine, PolicyRegistry, Matchable, LogFieldSelector};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create registry and load policies
    let registry = PolicyRegistry::new();
    let provider = FileProvider::new("policies.json");
    registry.subscribe(&provider)?;

    // Create engine and get snapshot
    let engine = PolicyEngine::new();
    let snapshot = registry.snapshot();

    // Evaluate a log record
    let log = MyLogRecord::new("Error: connection timeout", "ERROR");
    let result = engine.evaluate(&snapshot, &log).await?;

    match result {
        EvaluateResult::NoMatch => println!("Pass through"),
        EvaluateResult::Keep { policy_id, .. } => println!("Keep: {}", policy_id),
        EvaluateResult::Drop { policy_id } => println!("Drop: {}", policy_id),
        EvaluateResult::Sample { keep, .. } => println!("Sampled: {}", keep),
        EvaluateResult::RateLimit { allowed, .. } => println!("Rate limited: {}", allowed),
    }

    Ok(())
}

Core Concepts

Policy Registry

The PolicyRegistry manages policies from multiple providers and maintains an immutable snapshot for lock-free evaluation:

let registry = PolicyRegistry::new();

// Subscribe to a file-based provider (auto-reloads on changes)
let provider = FileProvider::new("policies.json");
registry.subscribe(&provider)?;

// Or register a custom provider
let handle = registry.register_provider();
handle.update(vec![policy1, policy2]);

// Get immutable snapshot for evaluation
let snapshot = registry.snapshot();

Policy Engine

The PolicyEngine evaluates logs against compiled policies using Hyperscan for pattern matching:

let engine = PolicyEngine::new();
let snapshot = registry.snapshot();

// Read-only evaluation
let result = engine.evaluate(&snapshot, &log).await?;

// Evaluation with transformations applied
let result = engine.evaluate_and_transform(&snapshot, &mut log).await?;

Evaluation Results

pub enum EvaluateResult {
    /// No policies matched - pass through unchanged
    NoMatch,
    /// Matched policy says keep
    Keep { policy_id: String, transformed: bool },
    /// Matched policy says drop
    Drop { policy_id: String },
    /// Matched policy says sample (percentage-based)
    Sample { policy_id: String, percentage: f64, keep: bool, transformed: bool },
    /// Matched policy says rate limit (count-based)
    RateLimit { policy_id: String, allowed: bool, transformed: bool },
}

Implementing the Traits

To evaluate your log types, implement the Matchable trait. For transformation support, also implement Transformable.

Matchable Trait

The Matchable trait provides zero-allocation field access for pattern matching:

use policy_rs::{Matchable, LogFieldSelector};
use policy_rs::proto::tero::policy::v1::LogField;

struct MyLogRecord {
    body: String,
    severity: String,
    attributes: HashMap<String, String>,
}

impl Matchable for MyLogRecord {
    fn get_field(&self, field: &LogFieldSelector) -> Option<&str> {
        match field {
            LogFieldSelector::Simple(log_field) => match log_field {
                LogField::Body => Some(&self.body),
                LogField::SeverityText => Some(&self.severity),
                _ => None,
            },
            LogFieldSelector::LogAttribute(key) => {
                self.attributes.get(key).map(|s| s.as_str())
            },
            LogFieldSelector::ResourceAttribute(key) => None,
            LogFieldSelector::ScopeAttribute(key) => None,
        }
    }
}

Transformable Trait

The Transformable trait enables field mutations when using evaluate_and_transform:

use policy_rs::{Transformable, LogFieldSelector};

impl Transformable for MyLogRecord {
    fn remove_field(&mut self, field: &LogFieldSelector) -> bool {
        match field {
            LogFieldSelector::LogAttribute(key) => {
                self.attributes.remove(key).is_some()
            },
            _ => false,
        }
    }

    fn redact_field(&mut self, field: &LogFieldSelector, replacement: &str) -> bool {
        match field {
            LogFieldSelector::LogAttribute(key) => {
                if self.attributes.contains_key(key) {
                    self.attributes.insert(key.clone(), replacement.to_string());
                    true
                } else {
                    false
                }
            },
            _ => false,
        }
    }

    fn rename_field(&mut self, from: &LogFieldSelector, to: &str, upsert: bool) -> bool {
        if let LogFieldSelector::LogAttribute(key) = from {
            if let Some(value) = self.attributes.remove(key) {
                if upsert || !self.attributes.contains_key(to) {
                    self.attributes.insert(to.to_string(), value);
                    return true;
                }
            }
        }
        false
    }

    fn add_field(&mut self, field: &LogFieldSelector, value: &str, upsert: bool) -> bool {
        match field {
            LogFieldSelector::LogAttribute(key) => {
                if upsert || !self.attributes.contains_key(key) {
                    self.attributes.insert(key.clone(), value.to_string());
                    true
                } else {
                    false
                }
            },
            _ => false,
        }
    }
}

Advanced Usage

Custom Policy Providers

Implement PolicyProvider to load policies from custom sources:

use policy_rs::{PolicyProvider, PolicyCallback, Policy, PolicyError};

struct MyProvider {
    // Your state here
}

impl PolicyProvider for MyProvider {
    fn load(&self, callback: &PolicyCallback) -> Result<(), PolicyError> {
        let policies = self.fetch_policies()?;
        callback.update(policies);
        Ok(())
    }
}

// Use with the registry
let registry = PolicyRegistry::new();
let provider = MyProvider::new();
registry.subscribe(&provider)?;

Policy Statistics

Track policy hit/miss rates and transform statistics:

let snapshot = registry.snapshot();

for entry in snapshot.iter() {
    let stats = entry.stats.snapshot();

    println!("Policy: {}", entry.policy.id());
    println!("  Matches: {} hits, {} misses", stats.match_hits, stats.match_misses);
    println!("  Remove: {} hits, {} misses", stats.remove.0, stats.remove.1);
    println!("  Redact: {} hits, {} misses", stats.redact.0, stats.redact.1);
    println!("  Rename: {} hits, {} misses", stats.rename.0, stats.rename.1);
    println!("  Add: {} hits, {} misses", stats.add.0, stats.add.1);
}

Multiple Providers

Combine policies from multiple sources:

let registry = PolicyRegistry::new();

// File-based policies
let file_provider = FileProvider::new("local-policies.json");
registry.subscribe(&file_provider)?;

// Programmatic policies
let handle = registry.register_provider();
handle.update(vec![
    create_emergency_drop_policy(),
    create_rate_limit_policy(),
]);

// All policies are merged in the snapshot
let snapshot = registry.snapshot();

Configuration-Based Providers

Use the config module to define providers in JSON/TOML configuration files. The ProviderConfig type is designed to be embedded in your application's config:

use policy_rs::config::{ProviderConfig, register_providers};
use policy_rs::PolicyRegistry;
use serde::Deserialize;

#[derive(Deserialize)]
struct AppConfig {
    service_name: String,
    policy_providers: Vec<ProviderConfig>,
}

// Parse your app config
let config: AppConfig = serde_json::from_str(r#"{
    "service_name": "my-app",
    "policy_providers": [
        {
            "id": "local",
            "type": "file",
            "path": "policies.json"
        },
        {
            "id": "remote",
            "type": "http",
            "url": "https://api.example.com/policies",
            "headers": [
                { "name": "Authorization", "value": "Bearer token123" }
            ],
            "poll_interval_secs": 60
        }
    ]
}"#)?;

// Register all providers at once
let registry = PolicyRegistry::new();
register_providers(&config.policy_providers, &registry)?;

Provider Config Format

Each provider configuration has a type field that determines the provider:

File Provider:

{
  "id": "local-policies",
  "type": "file",
  "path": "policies.json"
}

HTTP Provider (requires http feature):

{
  "id": "remote-policies",
  "type": "http",
  "url": "https://api.example.com/policies",
  "headers": [{ "name": "Authorization", "value": "Bearer token" }],
  "poll_interval_secs": 60,
  "content_type": "application/json"
}

gRPC Provider (requires grpc feature):

{
  "id": "grpc-policies",
  "type": "grpc",
  "endpoint": "https://grpc.example.com:443"
}

You can also parse just the provider list directly:

let providers: Vec<ProviderConfig> = serde_json::from_str(r#"[
    { "id": "file", "type": "file", "path": "policies.json" }
]"#)?;

Transform Order

When using evaluate_and_transform, transformations are applied in a fixed order:

1. Remove - Delete fields
2. Redact - Replace field values with placeholders
3. Rename - Rename fields to new keys
4. Add - Add new fields

Transforms from all matching policies are applied, not just the winning policy.

Policy Format

Policies are defined using the Tero Policy protobuf schema. Example JSON:

{
  "id": "drop-debug-logs",
  "name": "Drop Debug Logs",
  "enabled": true,
  "target": {
    "log": {
      "match": [
        {
          "logField": "SEVERITY_TEXT",
          "regex": "DEBUG|TRACE"
        }
      ],
      "keep": "none"
    }
  }
}

Keep Values

• "all" - Keep all matching logs
• "none" - Drop all matching logs
• "50%" - Sample 50% of matching logs
• "100/s" - Rate limit to 100 logs per second
• "1000/m" - Rate limit to 1000 logs per minute

Match Fields

• logField - Simple fields: BODY, SEVERITY_TEXT, TRACE_ID, SPAN_ID, etc.
• logAttribute - Log attributes by key
• resourceAttribute - Resource attributes by key
• scopeAttribute - Scope attributes by key

Match Types

• exact - Exact string match
• regex - Regular expression match
• exists - Field existence check

Examples

See the examples/ directory:

• basic_usage.rs - Load policies and evaluate logs
• transforms.rs - Apply log transformations
• multiple_providers.rs - Combine multiple policy sources
• custom_provider.rs - Implement a custom provider
• config_providers.rs - Configure providers via JSON config

Run examples with:

cargo run --example basic_usage
cargo run --example transforms
cargo run --example config_providers

License

Apache-2.0