Struct ChunkBatcher 

pub struct ChunkBatcher { /* private fields */ }

Implements an asynchronous batcher that coalesces PendingRows into Chunks based upon the thresholds defined in the associated ChunkBatcherConfig.

Batching vs. splitting

The batching process is triggered solely by time and space thresholds – whichever is hit first. This process will result in one big dataframe.

The splitting process will then run on top of that big dataframe, and split it further down into smaller Chunks. Specifically, the dataframe will be splits into enough Chunks so as to guarantee that:

• no chunk contains data for more than one entity path
• no chunk contains rows with different sets of timelines
• no chunk uses more than one datatype for a given component
• no chunk contains more rows than a pre-configured threshold if one or more timelines are unsorted

Multithreading and ordering

ChunkBatcher can be cheaply clone and used freely across any number of threads.

Internally, all operations are linearized into a pipeline:

• All operations sent by a given thread will take effect in the same exact order as that thread originally sent them in, from its point of view.
• There isn’t any well defined global order across multiple threads.

This means that e.g. flushing the pipeline (Self::flush_blocking) guarantees that all previous data sent by the calling thread has been batched and sent down the channel returned by ChunkBatcher::chunks; no more, no less.

Shutdown

The batcher can only be shutdown by dropping all instances of it, at which point it will automatically take care of flushing any pending data that might remain in the pipeline.

Shutting down cannot ever block.

Implementations

impl ChunkBatcher

pub fn new( config: ChunkBatcherConfig, hooks: BatcherHooks, ) -> ChunkBatcherResult<Self>

Creates a new ChunkBatcher using the passed in config.

The returned object must be kept in scope: dropping it will trigger a clean shutdown of the batcher.

pub fn push_chunk(&self, chunk: Chunk)

pub fn push_row(&self, entity_path: EntityPath, row: PendingRow)

Pushes a PendingRow down the batching pipeline.

This will computea the size of the row from the batching thread!

See ChunkBatcher docs for ordering semantics and multithreading guarantees.

pub fn flush_async(&self)

Initiates a flush of the pipeline and returns immediately.

This does not wait for the flush to propagate (see Self::flush_blocking). See ChunkBatcher docs for ordering semantics and multithreading guarantees.

pub fn flush_blocking(&self, timeout: Duration) -> Result<(), BatcherFlushError>

Initiates a flush the batching pipeline and waits for it to propagate.

See ChunkBatcher docs for ordering semantics and multithreading guarantees.

pub fn update_config(&self, config: ChunkBatcherConfig)

Updates the batcher’s configuration as far as possible.

pub fn chunks(&self) -> Receiver<Chunk>

Returns a shared channel in which are sent the batched Chunks.

Shutting down the batcher will close this channel.

See ChunkBatcher docs for ordering semantics and multithreading guarantees.

Trait Implementations

impl Clone for ChunkBatcher

fn clone(&self) -> ChunkBatcher

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.