tp-lexer: no_std filter expression engine for tracepoint-like events
tp-lexer parses and evaluates boolean filter expressions combining numeric and byte/string comparisons for event pipelines (eBPF / tracepoints / custom telemetry). It targets no_std (with alloc) and supports direct evaluation over raw byte buffers or map-based contexts.
Features
- Numeric operators:
== != < <= > >=(bit mask form:field & 0x10 == 0x10). - Byte/string operators:
== != ~(glob:* ? [..]with class negation[!a]/[^a]). - Unquoted barewords allowed:
comm != bash; quote when spaces or special chars present ("C Program"). - Precedence: parentheses > comparison >
&&>||(short-circuit evaluation). - Tri-state runtime: missing field =>
Unknown.Unknown && True=>True(true-neutral)Unknown || False=>False(false-neutral)- Top-level
Unknown=> treated asTrue(fail-open to avoid silent data loss).
- Signed decimal / hex integer literals such as
-1,0x10, and-(0x10). u64/u128/i128/isize/usizefields, plus theirNonZero*variants, are supported, but evaluation still uses an internali64; values outside thei64range are truncated with Rust's integer cast semantics before comparison.- Compile-time validation: unknown field or type mismatch produces a
ParseError. - Raw buffer evaluation (
BufContext) using schema offsets; no copying.
Schema Definition
Define schemas with the schema! macro. Each entry: "name" => (FieldType, offset, length).
Integer types implement the FieldClassifier trait exposing FIELD_TYPE. Bytes fields use FieldType::Bytes.
use ;
let schema = schema!;
let expr = "sig >= 10 && comm != bash";
let compiled = compile_with_schema.unwrap;
Map-Based Context Example
use BTreeMap;
use ;
let schema = schema!;
let compiled = compile_with_schema.unwrap;
let mut ctx = from;
assert!;
Raw Buffer Context Example
use ;
let schema = schema!;
let mut buf = ;
buf.copy_from_slice;
buf.copy_from_slice;
let ctx = new;
let compiled = compile_with_schema.unwrap;
assert!;
Glob Matching
let schema = schema!;
let compiled = compile_with_schema.unwrap;
let mut buf = ;
buf.copy_from_slice; // unused
buf.copy_from_slice;
let ctx = new;
assert!;
Bit Mask Example
let schema = schema!;
let compiled = compile_with_schema.unwrap;
let mut buf = ;
buf.copy_from_slice; // 0x30 & 0x10 == 0x10 -> true
let ctx = new;
assert!;
Errors & Semantics
Compile-time errors:
- Unknown field names
- Type mismatches (numeric op on bytes field; string op on integer field)
- Syntax errors (unrecognized tokens, unterminated string, lone operator characters)
Runtime semantics:
- Missing values =>
Unknowntri-state propagation - Top-level
Unknown=> treated astrue(fail-open) - Short-circuit evaluation limits unnecessary field access
- Wide integer fields (
u64/u128/i128/isize/usize) and theirNonZero*variants are cast to the engine's internali64representation before comparison, which may truncate high bits or flip sign for out-of-range values
Performance Notes
- Bit masking done after integer fetch; consider pre-masked fields if hot.
- Glob matcher is a simple recursive backtracking implementation; patterns dominated by leading
*may degrade performance—structure patterns to avoid pathological worst cases. - For high-throughput paths, prefer
BufContextto avoid string allocations.
Non-Goals / Current Omissions
- No regex (glob only)
- No arithmetic besides bit masking
- No user-defined functions
- No range syntax (
[a,b]) or set membership; can be emulated with chained comparisons / ORs
Testing
Operator coverage tests ensure every numeric and string comparison path, mask usage, and type mismatch branch is exercised (cargo test -p tp-lexer).
Build & Run
Tri-State Rationale
Fail-open semantics (Unknown => true) prevent accidental data loss when filters reference newly introduced fields not yet populated. If strict filtering is required, introduce an explicit presence indicator field and include it in the expression.