Module datafusion::datasource

DataFusion data sources: TableProvider and ListingTable

Re-exports

• pub use self::default_table_source::provider_as_source;
• pub use self::default_table_source::source_as_provider;
• pub use self::default_table_source::DefaultTableSource;
• pub use self::memory::MemTable;
• pub use self::provider::TableProvider;
• pub use self::view::ViewTable;

Modules

• avro_to_arrow
  This module contains code for reading Avro data into RecordBatches
• cte_worktable
  CteWorkTable implementation used for recursive queries
• default_table_source
  Default TableSource implementation used in DataFusion physical plans
• empty
  EmptyTable useful for testing.
• file_format
  Module containing helper methods for the various file formats See write.rs for write related helper methods
• function
  A table that uses a function to generate data
• listing
  A table that uses the ObjectStore listing capability to get the list of files to process.
• listing_table_factory
  Factory for creating ListingTables with default options
• memory
  MemTable for querying Vec<RecordBatch> by DataFusion.
• object_store
  ObjectStoreRegistry holds all the object stores at Runtime with a scheme for each store. This allows the user to extend DataFusion with different storage systems such as S3 or HDFS and query data inside these systems.
• physical_plan
  Execution plans that read file formats
• provider
  Data source traits
• schema_adapter
  SchemaAdapter and SchemaAdapterFactory to adapt file-level record batches to a table schema.
• stream
  TableProvider for stream sources, such as FIFO files
• streaming
  A simplified TableProvider for streaming partitioned datasets
• view
  View data source which uses a LogicalPlan as it’s input.

Enums

• TableType
  Indicates the type of this table for metadata/catalog purposes.

Functions

• get_statistics_with_limit
  Get all files as well as the file level summary statistics (no statistic for partition columns). If the optional limit is provided, includes only sufficient files. Needed to read up to limit number of rows. collect_stats is passed down from the configuration parameter on ListingTable. If it is false we only construct bare statistics and skip a potentially expensive call to multiunzip for constructing file level summary statistics.