Module datafusion::datasource

DataFusion data sources

Re-exports

pub use self::datasource::TableProvider;

pub use self::default_table_source::provider_as_source;

pub use self::default_table_source::source_as_provider;

pub use self::default_table_source::DefaultTableSource;

pub use self::memory::MemTable;

pub use self::view::ViewTable;

Modules

datasource

Data source traits

default_table_source

Default TableSource implementation used in DataFusion physical plans

empty

An empty plan that is usefull for testing and generating plans without mapping them to actual data.

file_format

Module containing helper methods for the various file formats

listing

A table that uses the ObjectStore listing capability to get the list of files to process.

memory

In-memory data source for presenting a Vec as a data source that can be queried by DataFusion. This allows data to be pre-loaded into memory and then repeatedly queried without incurring additional file I/O overhead.

object_store

ObjectStoreRegistry holds all the object stores at Runtime with a scheme for each store. This allows the user to extend DataFusion with different storage systems such as S3 or HDFS and query data inside these systems.

view

View data source which uses a LogicalPlan as it’s input.

Enums

TableType

Indicates the type of this table for metadata/catalog purposes.

Functions

get_statistics_with_limit

Get all files as well as the file level summary statistics (no statistic for partition columns). If the optional limit is provided, includes only sufficient files. Needed to read up to limit number of rows. TODO fix case where num_rows and total_byte_size are not defined (stat should be None instead of Some(0))