Struct QueryHandle

pub struct QueryHandle<E: StorageEngineLike> { /* private fields */ }

A handle to a dataframe query, ready to be executed.

Cheaply created via QueryEngine::query.

See QueryHandle::next_row or QueryHandle::into_iter.

Implementations

impl<E: StorageEngineLike> QueryHandle<E>

pub fn query(&self) -> &QueryExpression

The query used to instantiate this handle.

pub fn view_contents(&self) -> &SorbetColumnDescriptors

Describes the columns that make up this view.

See QueryExpression::view_contents.

pub fn selected_contents(&self) -> &[(usize, ColumnDescriptor)]

Describes the columns that make up this selection.

The extra usize is the index in Self::view_contents that this selection points to.

See QueryExpression::selection.

pub fn schema(&self) -> &ArrowSchemaRef

All results returned by this handle will strictly follow this Arrow schema.

Columns that do not yield any data will still be present in the results, filled with null values.

pub fn seek_to_row(&self, row_idx: usize)

Advance all internal cursors so that the next row yielded will correspond to row_idx.

Does nothing if row_idx is out of bounds.

Concurrency

Cursors are implemented using atomic variables, which means calling any of the seek_* while iteration is concurrently ongoing is memory-safe but logically undefined racy behavior. Be careful.

Performance

This requires going through every chunk once, and for each chunk running a binary search if the chunk’s time range contains the index_value.

I.e.: it’s pretty cheap already.

pub fn num_rows(&self) -> u64

How many rows of data will be returned?

The number of rows depends and only depends on the view contents. The selected contents has no influence on this value.

pub fn next_row(&self) -> Option<Vec<ArrayRef>>

Returns the next row’s worth of data.

The returned vector of Arrow arrays strictly follows the schema specified by Self::schema. Columns that do not yield any data will still be present in the results, filled with null values.

Each cell in the result corresponds to the latest locally known value at that particular point in the index, for each respective ColumnDescriptor. See QueryExpression::sparse_fill_strategy to go beyond local resolution.

Example:

while let Some(row) = query_handle.next_row() {
    // …
}

Pagination

Use Self::seek_to_row:

query_handle.seek_to_row(42);
for row in query_handle.into_iter().take(len) {
    // …
}

pub fn next_row_async(&self) -> impl Future<Output = Option<Vec<ArrayRef>>>

where E: 'static + Send + Clone,

Asynchronously returns the next row’s worth of data.

The returned vector of Arrow arrays strictly follows the schema specified by Self::schema. Columns that do not yield any data will still be present in the results, filled with null values.

Each cell in the result corresponds to the latest locally known value at that particular point in the index, for each respective ColumnDescriptor. See QueryExpression::sparse_fill_strategy to go beyond local resolution.

Example:

while let Some(row) = query_handle.next_row_async().await {
    // …
}

pub fn _next_row( &self, store: &ChunkStore, cache: &QueryCache, ) -> Option<Vec<ArrowArrayRef>>

pub fn next_row_batch(&self) -> Option<ArrowRecordBatch>

Calls Self::next_row and wraps the result in a ArrowRecordBatch.

Only use this if you absolutely need a ArrowRecordBatch as this adds a some overhead for schema validation.

See Self::next_row for more information.

pub async fn next_row_batch_async(&self) -> Option<ArrowRecordBatch>

where E: 'static + Send + Clone,

impl<E: StorageEngineLike> QueryHandle<E>

pub fn iter(&self) -> impl Iterator<Item = Vec<ArrowArrayRef>> + '_

Returns an iterator backed by Self::next_row.

pub fn into_iter(self) -> impl Iterator<Item = Vec<ArrowArrayRef>>

Returns an iterator backed by Self::next_row.

pub fn batch_iter(&self) -> impl Iterator<Item = ArrowRecordBatch> + '_

Returns an iterator backed by Self::next_row_batch.

pub fn into_batch_iter(self) -> impl Iterator<Item = ArrowRecordBatch>

Returns an iterator backed by Self::next_row_batch.