Struct DatasetState 

pub struct DatasetState {
    pub schema: DatasetSchema,
    pub data: Vec<RecordBatch>,
    pub arrow_schema: Arc<Schema>,
    pub index: HashMap<String, HashMap<String, Vec<u32>, RandomState>, RandomState>,
    pub lazy: bool,
}

Per-dataset state: schema metadata, the resident chunks, and the equality index built per the dataset’s [dataset.index] policy.

data is the dataset as a Vec<RecordBatch> — exactly the chunks produced by the underlying reader, after temporal columns are cast to Utf8. We deliberately do not call concat_batches to fuse them into one batch: on wide schemas (hundreds of columns) that transiently allocates a second full copy of the decoded Arrow data, pushing peak RSS to ~2× the resident size and OOM-killing the process at startup.

When lazy is true the dataset is not materialised: data is empty, index is empty, and every query is dispatched to DataFusion SQL against a registered ListingTable. arrow_schema still carries the inferred schema so discovery endpoints work.

Fields

schema: DatasetSchema

data: Vec<RecordBatch>

arrow_schema: Arc<Schema>

index: HashMap<String, HashMap<String, Vec<u32>, RandomState>, RandomState>

lazy: bool

Implementations

impl DatasetState

pub fn num_rows(&self) -> usize

Sum of num_rows() across all resident chunks. 0 for lazy datasets.