Skip to main content

Module driver

hyperi_rustlib::worker::engine

Module driver 

Source
Available on crate feature transport and crate feature worker-batch and (crate features worker-batch or worker-pool or worker) only.
Expand description

§Unified WorkBatch engine driver

The single run loop that the four legacy loops (run / run_raw / run_async / run_raw_async) collapsed into when the spine flipped to WorkBatch (Task 0.7b). It drives the canonical currency – WorkBatch – through one block at a time:

  recv(max) -> WorkBatch        (recv now yields a WorkBatch natively)
    -> apply_workbatch_dlq_policy  (route/discard/reject inline-DLQ entries)
    -> lease_ingress_batch      (memory accounting on the block's bytes)
    -> process(WorkBatch)       (transforms parse ON DEMAND via codec::parse)
    -> sink(&out_batch).await   (async send of the whole block)
    -> commit per CommitMode    (at-least-once, AFTER the block is sent)

§Why tokens live on the batch, not the record

WorkBatch::commit_tokens are the INPUT source acks. They are decoupled from records.len(), so a process that fans N records out to 2N (or collapses them) does NOT disturb the source acks. The driver commits EXACTLY the input tokens after the whole out-batch is sent – never 2N, never per output record. That invariant is the data-plane core; the fan-out commit-correctness test proves it.

§Two parse modes (the hybrid)

  • run_workbatch – the DEFAULT. The driver does NOT pre-parse. A transform that needs a field calls codec::parse on demand. Pass-through apps (receiver, raw forwarders) never pay a parse.

  • run_workbatch_parsed – opt-in for hot pipelines. The driver pre-parses the whole block via codec::parse (SIMD JSON / native MsgPack) on the worker pool and hands the process closure a ParsedBatch – records + their aligned ParsedPayloads + a shared FieldInterner for hot routing-field dedup. This keeps the batch-parse + interner throughput win for apps that opt in.

process_mid_tier, process_raw and ParsedMessage remain for the in-process (non-run-loop) callers; only the four legacy run loops were removed by the 0.7b flip.

Structs§

ParsedBatch
A pre-parsed block for the opt-in run_workbatch_parsed hot path.

Enums§

CommitMode
When the driver commits the input source acks.