Skip to main content

Module super_table

simd_kernels::structs::chunked

Module super_table 

Source
Expand description

§SuperTable - Holds multiple Tables for chunked data partitioning, streaming + fast memIO

SuperTable groups multiple Table batches under a shared schema.

§Purpose

  • Treats an ordered sequence of Table batches as one dataset.
  • Preserves per-batch independence while allowing unified export over Arrow FFI.
  • Enables both bounded and unbounded (Live / streaming) workflows.

§Behaviour

  • All batches must have identical column definitions (Field metadata).
  • Row counts may differ between batches, but not between inner arrays.
  • When sent via Arrow FFI, exposed as a single logical table.
  • Supports concatenation into a materialised Table on demand.

§Typical Uses

  • Partitioned storage readers (e.g., multiple Arrow IPC files).
  • Streaming ingestion into append-only datasets.
  • Windowed or mini-batch analytics.
  • Incremental build-up of tables for later unification.

Structs§

SuperTable
SuperTable