Struct MessageBatchConfig 

pub struct MessageBatchConfig {
    pub max_chunks: usize,
    pub flush_interval_ms: Option<u64>,
}

Configuration for batching LLM streaming chunks.

Controls how token chunks are accumulated before forwarding to stream consumers. Batching reduces overhead for high-volume token streaming by coalescing small chunks into fewer, larger deliveries.

Fields

max_chunks: usize

Maximum number of chunks to accumulate before flushing.

When this many chunks are collected, they are flushed immediately regardless of the time threshold.

flush_interval_ms: Option<u64>

Maximum time in milliseconds to wait before flushing.

If this duration elapses without reaching max_chunks, the accumulated chunks are flushed. A value of None disables time-based flushing.

Implementations

impl MessageBatchConfig

pub const fn new(max_chunks: usize, flush_interval_ms: Option<u64>) -> Self

Create a new batch config with the specified parameters.

pub const fn no_batching() -> Self

Create config with no batching (flush every chunk immediately).

Trait Implementations

impl Clone for MessageBatchConfig

fn clone(&self) -> MessageBatchConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MessageBatchConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for MessageBatchConfig

fn default() -> Self

Returns the “default value” for a type.