[][src]Crate multistream_batch

This Rust library provides different types and implementations of batching algorithms.

Batching is based on collecting items and flushing them all together when batch has reached some limit or when manually flushed. This makes all items collected in single batch available at once for further processing (e.g. batch insert into a database).

This implementations will construct batches based on:

  • maximum number of items collected,
  • maximum time duration since first item was collected by the batch,
  • calling one of the batch consuming methods,
  • sending flush command between batch items (channel based batches).

See sub modules for documentation of available algorithms.

Modules

buf_batch

This module provides BufBatch that will buffer items until batch is ready and provide them in one go using Drain iterator.

channel

Batch implementations that use channels and threads to support simultaneously receiving items and awaiting on timeouts.

multi_buf_batch

This module provides MultiBufBatch that will buffer items into multiple internal batches based on batch stream key until one of the batches is ready and provides this items in one go along with the batch stream key using Drain iterator.