Skip to main content

Module super_array

simd_kernels::structs::chunked

Module super_array 

Source
Expand description

§SuperArray - Holds multiple arrays for chunked data partitioning, streaming + fast memIO

Contains SuperArray, a higher-order container representing a logical column split into multiple immutable Array chunks with optional shared field metadata.

§Overview

  • Equivalent to Apache Arrow’s ChunkedArray.
  • Stores an ordered list of Array segments with shared field metadata.
  • Chunk lengths may vary.
  • A solid fit for append-only patterns, partitioned storage, and streaming data ingestion.

§Field Metadata

  • Field metadata is stored once at the SuperArray level, not per-chunk.
  • Use from_arrays() when you don’t need field metadata (e.g., Dam consolidation)
  • Use from_arrays_with_field() when field metadata is required

Structs§

SuperArray
SuperArray

Enums§

RechunkStrategy
Strategy for rechunking arrays and tables.