Struct BackgroundConfig 

pub struct BackgroundConfig {
    pub num_shards: usize,
    pub linger_time: IggyDuration,
    pub error_callback: Arc<Box<dyn ErrorCallback + Send + Sync>>,
    pub sharding: Box<dyn Sharding + Send + Sync>,
    pub batch_size: usize,
    pub batch_length: usize,
    pub failure_mode: BackpressureMode,
    pub max_buffer_size: IggyByteSize,
    pub max_in_flight: usize,
}

The Apache Iggy producer/consumer escape hatch: same types Topic::iggy_producer/Topic::iggy_consumer/Topic::iggy_consumer_group return, and the stored message a failed publish hands back, re-exported so reaching for them doesn’t need an iggy dependency of your own. Configuration for a producer that sends messages in the background.

A background producer passes every non-empty send to a ProducerDispatcher. The dispatcher returns once the batch is queued, and one of its worker Shards writes the batch later. This type controls how many workers exist, which worker receives a batch, when workers flush, how many bytes may be queued or in flight, and where write failures are reported.

Defaults

Field	Default	Controls
num_shards	1	how many worker queues exist
sharding	OrderedSharding	which worker a batch is queued on
batch_size	1 MiB	flush once a worker holds this many bytes
batch_length	1000	flush once a worker holds this many queued sends
linger_time	1 ms	how long a worker holds a non-empty buffer before flushing it
max_buffer_size	32 MiB	bytes the whole producer may hold
failure_mode	BackpressureMode::Block	what a send does when that budget is full
max_in_flight	1	requests being written at once
error_callback	LogErrorCallback	what happens to a failed write

The default OrderedSharding strategy preserves dispatch order for each stream/topic pair by routing that pair to one worker. That worker awaits each request, including its retries, before starting the next. BalancedSharding can route consecutive batches for one destination to different workers and therefore gives up that ordering. max_in_flight only controls how many workers may write concurrently. It does not make a single worker process more than one request at a time.

Zero values

0 disables the batch_size and batch_length flush thresholds. A zero linger_time flushes as soon as the worker picks up a send. A zero max_buffer_size is treated as unbounded. A zero max_in_flight uses Semaphore::MAX_PERMITS, and a zero num_shards is read as one worker.

Examples

use iggy::clients::producer_config::BackpressureMode;
use iggy::prelude::*;
use std::time::Duration;

// Ordered and bounded, as described above.
let ordered = BackgroundConfig::builder().build();

// Throughput: four workers share one topic, flushing at 4 MiB, with a 256 MiB byte budget.
// Up to 4 requests can be in flight at the same time, one per worker.
let fast = BackgroundConfig::builder()
    .num_shards(4)
    .sharding(Box::new(BalancedSharding::default()))
    .batch_size(4 * 1024 * 1024)
    .max_buffer_size(IggyByteSize::from(256 * 1024 * 1024))
    .max_in_flight(4)
    .build();

// Latency: flush within 5 ms or after 50 queued sends. Do not wait for byte-budget capacity.
// The bounded per-worker channel can still make dispatch wait when its 256 slots are occupied.
let responsive = BackgroundConfig::builder()
    .linger_time(IggyDuration::new(Duration::from_millis(5)))
    .batch_length(50)
    .failure_mode(BackpressureMode::FailImmediately)
    .build();

Fields

num_shards: usize

Number of worker Shards the dispatcher runs, each with a queue of its own.

Every batch is routed to exactly one worker by sharding, and each worker buffers and writes independently.

0 is read as one worker.

linger_time: IggyDuration

Upper bound on how long a worker holds a non-empty buffer before flushing it.

The window starts when a send enters an empty buffer, so an idle worker does not wake up. A worker flushes as soon as any of linger_time, batch_length or batch_size is reached. Lowering it reduces the time the write is delayed at the price of smaller writes. 0 flushes as soon as the worker picks up a send.

Note that IggyDuration::from reads a plain number as microseconds, so the default of 1000 is 1 ms.

error_callback: Arc<Box<dyn ErrorCallback + Send + Sync>>

Where a background write that failed ends up.

The dispatcher runs one task that owns this callback. A worker invokes it with an ErrorCtx when its backend returns IggyError::ProducerSendFailed. Other error variants from a custom backend are logged by the worker without invoking this callback. The context contains the cause, destination, unconfirmed tail, and confirmations returned for earlier chunks.

The default LogErrorCallback logs the failure and drops the messages with the context. Implement ErrorCallback with your own logic to keep them.

sharding: Box<dyn Sharding + Send + Sync>

Picks the worker a batch is queued on, out of num_shards.

The default OrderedSharding hashes stream and topic, so every batch for one topic queues on one worker and keeps the order it was dispatched in. BalancedSharding hands them out round-robin, which lets a single topic occupy all workers but gives up that order. Implement Sharding to build your own logic for picking a Shard.

batch_size: usize

Flush threshold in bytes buffered on one worker.

Sends accumulate until their reported sizes reach or exceed this value. The threshold is checked after a send is added, so it is a flush trigger rather than a hard size ceiling. 0 disables the threshold and leaves batch_length and linger_time to trigger the flush.

This is a per-worker flush trigger. It is separate from max_buffer_size, which caps the producer as a whole.

batch_length: usize

Flush threshold in number of queued batches on one worker.

Counts the queued sends a worker holds, not the individual messages inside them. A worker flushes after this many dispatches have been routed to it. 0 disables the threshold.

failure_mode: BackpressureMode

What a send does once max_buffer_size is exhausted.

max_buffer_size: IggyByteSize

Upper bound for the bytes buffered or in flight across all shards. Bytes remain charged until the corresponding write completes. IggyByteSize::from(0) means unlimited. A nonzero value greater than Semaphore::MAX_PERMITS makes ProducerDispatcher::new panic.

max_in_flight: usize

Upper bound on the requests being written concurrently, shared by all workers.

A worker takes one permit before each request and holds it until the write returns. This bounds write concurrency across the producer rather than per worker, and it does not bound queued bytes.

Each worker still sends sequentially regardless of this value. Raising it only lets different workers write concurrently. Per-destination order is therefore governed by sharding: OrderedSharding keeps one destination on one sequential worker, while strategies that spread a destination across workers may reorder it. A nonzero value greater than Semaphore::MAX_PERMITS makes ProducerDispatcher::new panic.

Implementations

impl BackgroundConfig

pub fn builder() -> BackgroundConfigBuilder

Create an instance of BackgroundConfig using the builder syntax

Trait Implementations

impl Debug for BackgroundConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.