Module worker 

Available on crate features worker-batch or worker-pool or worker only.

Adaptive worker pool and batch processing framework.

Two layers:

• Generic: AdaptiveWorkerPool provides CPU-saturating parallelism via rayon (CPU-bound) + tokio (async I/O), with reactive pressure-based scaling. Useful for any workload – not DFE-specific.

• Opinionated: BatchProcessor trait + BatchPipeline provide a structured parallel-then-sequential pipeline for DFE apps. Apps implement BatchProcessor for their domain; the pipeline handles stats, scaling, and batch orchestration. PipelineStats provides common atomic counters.

Quick Start

use hyperi_rustlib::worker::{AdaptiveWorkerPool, WorkerPoolConfig};

let pool = AdaptiveWorkerPool::from_cascade("worker_pool")?;
pool.register_metrics(&metrics_manager);
pool.start_scaling_loop(shutdown_token.clone());

// CPU-bound parallel transform (rayon)
let results = pool.process_batch(&messages, |msg| {
    parse_and_transform(msg)
});

// Async parallel enrichment (tokio)
let enriched = pool.fan_out_async(&items, |item| async move {
    enrich(item).await
}).await;

Re-exports

pub use engine::BatchEngine;worker-batch

pub use engine::BatchProcessingConfig;worker-batch

pub use engine::FieldInterner;worker-batch

pub use engine::MessageMetadata;worker-batch

pub use engine::ParsedMessage;worker-batch

pub use engine::PreRouteFilterConfig;worker-batch

pub use engine::CommitMode;transport and worker-batch

pub use engine::EngineError;transport and worker-batch

pub use engine::FilterDlqPolicy;transport and worker-batch

pub use engine::ParsedBatch;transport and worker-batch

Modules

engineworker-batch

ndjsonworker-batch

NDJSON (newline-delimited JSON) batch processing utilities.

Structs

AccumulatorConfig

Accumulator configuration.

AccumulatorFull

Error when the accumulator channel is full (backpressure).

AdaptiveWorkerPool

Adaptive worker pool with hybrid rayon (CPU) + tokio (async I/O) execution.

BatchAccumulator

Push handle – cloneable, used by producers to send items into the accumulator.

BatchDrainer

Drain handle – used by a single consumer to receive batches.

BatchPipeline

Orchestrates parallel batch processing via AdaptiveWorkerPool.

FanOutPolicy

Policy for AdaptiveWorkerPool::fan_out_async_with_policy.

PipelineStats

Common DFE pipeline statistics with atomic counters.

PipelineStatsSnapshot

Immutable snapshot of pipeline stats.

ScalingDecision

Result of the watermark scaling algorithm.

ScalingInput

Inputs to the watermark scaling algorithm.

WorkerPoolConfig

Configuration for the adaptive worker pool.

Enums

FanOutResult

Per-item outcome from AdaptiveWorkerPool::fan_out_async_with_policy.

Traits

BatchProcessor

Trait for parallel-safe message processing.

Functions

records_into_work_batch

Drain accumulated Records into a WorkBatch for push-ingest sources.