Crate bytecode_filter 

bytecode-filter

A fast bytecode-compiled filter engine for delimiter-separated records.

Filters are expressed in a small DSL, compiled to bytecode at startup, and evaluated with zero allocations in the hot path.

Features

• Zero-copy evaluation: Records are split into fields without copying
• SIMD-accelerated string matching: Uses memchr for fast substring search
• Precompiled regex: Regex patterns are compiled once at startup
• Key-value extraction: Extract and match key-value pairs from record fields
• Random sampling: Built-in rand(N) for probabilistic filtering

Example

use bytecode_filter::{compile, ParserConfig};
use bytes::Bytes;

// Define your record schema
let mut config = ParserConfig::default();
config.set_delimiter(",");
config.add_field("LEVEL", 0);
config.add_field("CODE", 1);
config.add_field("BODY", 2);

// Compile a filter expression
let filter = compile(r#"LEVEL == "error" AND CODE == "500""#, &config).unwrap();

// Evaluate against records
let record = Bytes::from("error,500,internal failure");
assert!(filter.evaluate(record));

let record = Bytes::from("info,200,ok");
assert!(!filter.evaluate(record));

Filter Syntax

Basic Operations

# Boolean literals
true
false

# Random sampling (returns true 1/N of the time)
rand(100)    # 1% sample
rand(2)      # 50% sample

# Payload-wide operations (match against the entire record)
payload contains "error"
payload starts_with "ERROR:"
payload ends_with ".json"
payload == "exact match"
payload matches "error_[0-9]+"

Field Operations

# Equality
STATUS == "active"
STATUS != "deleted"

# Set membership
LEVEL in {"error", "warn", "fatal"}

# String matching
PATH contains "/api/"
PATH starts_with "GET"
PATH matches "/api/v[0-9]+/.*"

# Case-insensitive
METHOD icontains "post"
LEVEL iequals "Error"

# Empty checks
NOTES is_empty
NOTES not_empty

Key-Value Extraction

For fields that contain key-value data (e.g., HTTP headers, metadata), you can extract individual values:

HEADERS.header("Content-Type") == "application/json"
HEADERS.header("Authorization") contains "Bearer"
HEADERS.header("X-Request-Id") exists

Boolean Logic

# AND, OR, NOT
LEVEL == "error" AND CODE == "500"
LEVEL == "warn" OR LEVEL == "error"
NOT LEVEL == "debug"

# Parentheses for grouping
(LEVEL == "error" OR LEVEL == "warn") AND BODY not_empty

Custom Schemas

Fields and delimiters are fully configurable via ParserConfig:

use bytecode_filter::ParserConfig;

let mut config = ParserConfig::default();
config.set_delimiter("\t");       // tab-separated
config.add_field("HOST", 0);
config.add_field("LEVEL", 1);
config.add_field("MESSAGE", 2);

Alternatively, use filter files with inline directives:

@delimiter = "\t"
@field HOST = 0
@field LEVEL = 1
@field MESSAGE = 2

LEVEL == "error" AND MESSAGE contains "timeout"

Structs

CompiledFilter

A compiled filter ready for evaluation.

Lexer

Lexer for filter expressions.

Parser

Parser for filter expressions.

ParserConfig

Parser configuration with field mappings.

PayloadParts

Zero-copy payload splitter.

Enums

CompileError

Compilation error types.

Expr

AST node for filter expressions.

LexError

Lexer error types.

LoadError

Error type for filter loading.

Opcode

Bytecode opcodes for the filter VM.

ParseError

Parser error types.

Token

Token types for filter expressions.

Constants

MAX_PARTS

Maximum number of parts supported.

Functions

compile

Compile a filter expression string into a CompiledFilter.

compile_expr

Compile a pre-parsed AST into a CompiledFilter.

extract_header_value

Extract an HTTP header value from a headers blob.

load_filter_file

Load and compile a filter from a file.

load_filter_string

Load and compile a filter from a string.

parse

Parse a filter expression string.

reset_rand_counter

Reset the random counter (for testing).