Struct DataBackendSession 

pub struct DataBackendSession {
    pub chunk_size: usize,
    pub runtime: Handle,
    /* private fields */
}

Provides a DataFusion session and registers DataFusion queries.

The session is used to register data sources and make queries on them. A query returns a Chunk of Arrow records. It is decoded and converted into a Vec of data by types that implement DecodeDataFromRecordBatch.

Fields

chunk_size: usize

runtime: Handle

Implementations

impl DataBackendSession

pub fn new(chunk_size: usize) -> Self

Creates a new DataBackendSession instance.

pub fn register_object_store( &mut self, url: &Url, object_store: Arc<dyn ObjectStore>, )

Register an object store with the session context

pub fn register_object_store_from_uri( &mut self, uri: &str, storage_options: Option<AHashMap<String, String>>, ) -> Result<()>

Register an object store with the session context from a URI with optional storage options.

Errors

Returns an error if the object store URI cannot be normalized or the backend cannot be created.

pub fn write_data<T: EncodeToRecordBatch>( data: &[T], metadata: &AHashMap<String, String>, stream: &mut dyn WriteStream, ) -> Result<(), DataStreamingError>

Writes encoded data to a streaming sink.

Errors

Returns an error if Arrow encoding or stream writing fails.

pub fn add_file<T>( &mut self, table_name: &str, file_path: &str, sql_query: Option<&str>, custom_type_name: Option<&str>, ) -> Result<()>

where T: DecodeDataFromRecordBatch,

Registers a Parquet file and adds a batch stream for decoding.

The caller must specify T to indicate the kind of data expected. table_name is the logical name for queries; file_path is the Parquet path; sql_query defaults to SELECT * FROM {table_name} ORDER BY ts_init if None.

When custom_type_name is Some, it is merged into each batch’s schema metadata before decoding (as type_name). Use this for custom data when Parquet/DataFusion does not preserve schema metadata so the decoder can look up the type in the registry.

The file data must be ordered by the ts_init in ascending order for this to work correctly.

Errors

Returns an error if parquet registration, SQL planning, stream execution, or data decoding setup fails.

pub fn collect_query_batches( &mut self, table_name: &str, file_path: &str, sql_query: Option<&str>, ) -> Result<Vec<RecordBatch>>

Registers a Parquet file and executes a query, returning the raw record batches.

Errors

Returns an error if parquet registration, SQL planning, stream execution, or batch collection fails.

pub fn get_query_result(&mut self) -> QueryResult

pub fn clear_registered_tables(&mut self)

Clears all registered tables and batch streams.

This is useful when the underlying files have changed and we need to re-register tables with updated data.