Struct LookupJoinOperator 

pub struct LookupJoinOperator { /* private fields */ }

Lookup join operator that enriches streaming events with external table data.

This operator caches lookup results in the state store to minimize external lookups. The cache uses processing-time-based TTL for expiration.

Async Lookup Handling

Since the Operator trait is synchronous, async table lookups must be handled externally. Use LookupJoinOperator::pending_lookups to get keys that need to be looked up, and LookupJoinOperator::provide_lookup to provide the results.

For simple synchronous use cases, use LookupJoinOperator::process_with_lookup which accepts a closure for lookups.

Implementations

impl LookupJoinOperator

pub fn new(config: LookupJoinConfig) -> Self

Creates a new lookup join operator.

pub fn with_id(config: LookupJoinConfig, operator_id: String) -> Self

Creates a new lookup join operator with explicit ID.

pub fn config(&self) -> &LookupJoinConfig

Returns the configuration.

pub fn metrics(&self) -> &LookupJoinMetrics

Returns the metrics.

pub fn reset_metrics(&mut self)

Resets the metrics.

pub fn pending_lookups(&self) -> &[Vec<u8>]

Returns keys that need external lookup.

After processing events, check this for keys that weren’t in the cache. Provide lookup results via provide_lookup.

pub fn provide_lookup( &mut self, key: &[u8], result: Option<&RecordBatch>, ctx: &mut OperatorContext<'_>, ) -> OutputVec

Provides a lookup result for a pending key.

Call this after performing the async lookup to complete processing of pending events.

Returns

Output events that can now be emitted after the lookup result is available.

pub fn process_with_lookup<F>( &mut self, event: &Event, ctx: &mut OperatorContext<'_>, lookup_fn: F, ) -> OutputVec

where F: FnOnce(&[u8]) -> Option<RecordBatch>,

Processes an event with a synchronous lookup function.

This is a convenience method for cases where lookups can be done synchronously (e.g., from an in-memory table).

Trait Implementations

impl Operator for LookupJoinOperator

fn process(&mut self, event: &Event, ctx: &mut OperatorContext<'_>) -> OutputVec

Process an incoming event

fn on_timer(&mut self, timer: Timer, ctx: &mut OperatorContext<'_>) -> OutputVec

Handle timer expiration

fn checkpoint(&self) -> OperatorState

Checkpoint the operator’s state

fn restore(&mut self, state: OperatorState) -> Result<(), OperatorError>

Restore from a checkpoint