Skip to main content

estimate_bytes_per_row

Function estimate_bytes_per_row 

Source
pub fn estimate_bytes_per_row(data_type: &DataType) -> f64
Expand description

Estimate the number of bytes per row for a given Arrow data type.

For fixed-width types this is exact. For variable-width types (strings, binary, lists) a rough default is used. The estimate is used as a starting point when batch_size_bytes is set; a post-decode feedback loop corrects it after the first batch.

This estimate ignores validity bitmaps at the moment. We can’t infer their presence simply from the data_type and their impact is probably fairly negligible. Returns a schema-based estimate of the decoded bytes per row for data_type.

Fixed-width types are exact. Variable-width types (strings, lists, etc.) use heuristic constants. This estimate is used both in batch-size planning and as a fallback for V1 files that lack structural decoders.