hyperi-rustlib 2.8.6

There's plenty of sage advice out there about how to run Rust services in production at scale — config cascades, structured logging, masking secrets, multi-backend secrets management, Prometheus, OpenTelemetry, Kafka transports, tiered disk-spillover sinks, adaptive worker pools, graceful shutdown — but almost none of it as code you can just install and use. This is that code. Opinionated, drop-in, working out of the box. The patterns from blog posts, watercooler chats and beers with your Google mates as actual library — not a framework you assemble from twenty crates and 8 weeks of munging.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

// Project:   hyperi-rustlib
// File:      src/worker/mod.rs
// Purpose:   Adaptive worker pool with hybrid rayon + tokio execution
// Language:  Rust
//
// License:   BUSL-1.1
// Copyright: (c) 2026 HYPERI PTY LIMITED

//! Adaptive worker pool and batch processing framework.
//!
//! Two layers:
//!
//! - **Generic:** [`AdaptiveWorkerPool`] provides CPU-saturating parallelism via
//!   rayon (CPU-bound) + tokio (async I/O), with reactive pressure-based scaling.
//!   Useful for any workload -- not DFE-specific.
//!
//! - **Opinionated:** [`BatchProcessor`] trait + [`BatchPipeline`] provide a
//!   structured parallel-then-sequential pipeline for DFE apps. Apps implement
//!   `BatchProcessor` for their domain; the pipeline handles stats, scaling,
//!   and batch orchestration. [`PipelineStats`] provides common atomic counters.
//!
//! ## Quick Start
//!
//! ```rust,ignore
//! use hyperi_rustlib::worker::{AdaptiveWorkerPool, WorkerPoolConfig};
//!
//! let pool = AdaptiveWorkerPool::from_cascade("worker_pool")?;
//! pool.register_metrics(&metrics_manager);
//! pool.start_scaling_loop(shutdown_token.clone());
//!
//! // CPU-bound parallel transform (rayon)
//! let results = pool.process_batch(&messages, |msg| {
//!     parse_and_transform(msg)
//! });
//!
//! // Async parallel enrichment (tokio)
//! let enriched = pool.fan_out_async(&items, |item| async move {
//!     enrich(item).await
//! }).await;
//! ```

mod accumulator;
mod batch;
mod config;
#[cfg(feature = "worker-batch")]
pub mod engine;
pub(crate) mod metrics;
#[cfg(feature = "worker-batch")]
pub mod ndjson;
mod pool;
pub(crate) mod scaler;
mod stats;

#[cfg(feature = "transport")]
pub use accumulator::records_into_work_batch;
pub use accumulator::{AccumulatorConfig, AccumulatorFull, BatchAccumulator, BatchDrainer};
pub use batch::{BatchPipeline, BatchProcessor};
pub use config::WorkerPoolConfig;
#[cfg(feature = "worker-batch")]
pub use engine::{
    BatchEngine, BatchProcessingConfig, FieldInterner, MessageMetadata, ParsedMessage,
    PreRouteFilterConfig,
};
#[cfg(all(feature = "worker-batch", feature = "transport"))]
pub use engine::{CommitMode, EngineError, FilterDlqPolicy, ParsedBatch};
pub use pool::{AdaptiveWorkerPool, FanOutPolicy, FanOutResult};
pub use scaler::{ScalingDecision, ScalingInput};
pub use stats::{PipelineStats, PipelineStatsSnapshot};