pub struct StreamingAggregator { /* private fields */ }Expand description
Constant-memory aggregation state that batches are pushed into.
Aggregation does not need to see its input all at once: every supported
function has an incremental form, so the state is O(groups × fields) and
each batch can be dropped as soon as it has been folded in. Holding the
input instead would make a SELECT avg(x) FROM m over a week of 1 s data
cost gigabytes for a single output row.
Rows must carry the group-by and field columns by name; a batch missing a column simply contributes nothing for it, which is how schema evolution across segments is tolerated.
Implementations§
Source§impl StreamingAggregator
impl StreamingAggregator
Sourcepub fn new(
functions: &[AggFn],
field_columns: &[&str],
group_by: &[&str],
) -> Self
pub fn new( functions: &[AggFn], field_columns: &[&str], group_by: &[&str], ) -> Self
Create an aggregator for functions over field_columns, grouped by
group_by (empty for a whole-dataset aggregate).
Sourcepub fn push(
&mut self,
batch: &RecordBatch,
memory_tracker: Option<&MemoryTracker>,
) -> Result<()>
pub fn push( &mut self, batch: &RecordBatch, memory_tracker: Option<&MemoryTracker>, ) -> Result<()>
Fold one batch into the running state. The batch may be dropped after.
§Errors
Returns an error if the optional memory_tracker budget is exceeded by
newly discovered groups.
Sourcepub fn finish(
self,
memory_tracker: Option<&MemoryTracker>,
) -> Result<RecordBatch>
pub fn finish( self, memory_tracker: Option<&MemoryTracker>, ) -> Result<RecordBatch>
Finalise the accumulated state into a result batch.
§Errors
Returns an error if the result batch cannot be built or the optional
memory_tracker budget is exceeded.