Crate multistream_batch

Implementations of batching algorithms.

Batching works by accumulating items and later automatically flushing them all together when the batch has reached a limit. All items collected in the single batch are available at once for further processing (e.g. batch insert into a database).

These implementations will construct batches based on:

• limit of the number of items collected in a batch,
• limit of time duration since the first item appended to the batch,
• calling one of the batch consuming methods,
• sending flush command between batch items (channel-based implementations).

See sub modules for documentation of available algorithms. !

Modules

buf_batch

This module provides BufBatch that will buffer items until the batch is ready and provide them in one go using Drain iterator.

channel

Batch implementations that use channels and threads to support simultaneously receiving items and awaiting on timeouts.

multi_buf_batch

This module provides MultiBufBatch that will buffer items into multiple internal batches based on batch stream key until one of the batches is ready. Then it provides accumulated items in one go along with the batch stream key using Drain iterator.