Struct Connector 

pub struct Connector { /* private fields */ }

Bundles a CatalogProvider with TableReaders for convenient use.

The Connector is the primary entry point for interacting with an external catalog. It delegates metadata operations to the catalog provider and data reading to the appropriate table reader based on the table’s data format.

Storage Options

Cloud storage credentials (S3, Azure, GCS) are passed via storage_options and forwarded to each TableReader during table registration.

Extensibility

• Swap the catalog: pass a different CatalogProvider (e.g., AWS Glue).
• Add formats: pass additional TableReaders (e.g., Iceberg).

Example

// let connector = Connector::new(catalog, readers)
//     .with_storage_options(HashMap::from([
//         ("aws_access_key_id".into(), "...".into()),
//         ("aws_secret_access_key".into(), "...".into()),
//     ]));

Implementations

impl Connector

pub fn new( catalog: Arc<dyn CatalogProvider>, readers: Vec<Arc<dyn TableReader>>, ) -> Connector

Create a new connector with the given catalog provider and table readers.

pub fn with_storage_options(self, options: HashMap<String, String>) -> Connector

Set storage options for cloud storage access (S3, Azure, GCS).

Common keys:

• S3: aws_access_key_id, aws_secret_access_key, aws_region
• Azure: azure_storage_account_name, azure_storage_account_key
• GCS: google_service_account_path

pub fn storage_options(&self) -> &HashMap<String, String>

Get the current storage options.

pub fn catalog(&self) -> &dyn CatalogProvider

Get a reference to the underlying catalog provider.

pub fn reader_for(&self, format: &DataSourceFormat) -> Option<&dyn TableReader>

Find a table reader that supports the given data format.

pub fn readers(&self) -> &[Arc<dyn TableReader>]

List all available table readers.

pub async fn list_catalogs(&self) -> Result<Vec<CatalogInfo>, CatalogError>

pub async fn get_catalog(&self, name: &str) -> Result<CatalogInfo, CatalogError>

pub async fn list_schemas( &self, catalog_name: &str, ) -> Result<Vec<SchemaInfo>, CatalogError>

pub async fn get_schema( &self, catalog_name: &str, schema_name: &str, ) -> Result<SchemaInfo, CatalogError>

pub async fn list_tables( &self, catalog_name: &str, schema_name: &str, ) -> Result<Vec<TableInfo>, CatalogError>

pub async fn get_table( &self, catalog_name: &str, schema_name: &str, table_name: &str, ) -> Result<TableInfo, CatalogError>

pub async fn register_schema( &self, ctx: &SessionContext, catalog_name: &str, schema_name: &str, ) -> Result<Vec<(String, Arc<Schema>)>, CatalogError>

Register all tables from a catalog schema into a DataFusion SessionContext.

For each table:

1. Retrieves full table metadata (including columns) from the catalog.
2. Converts columns to an Arrow schema.
3. Finds an appropriate TableReader for the table’s data format.
4. Registers the table in the session context with storage options for cloud access.

If no reader matches the table’s format, falls back to registering an empty MemTable with the correct schema (schema-only, for planning).

Individual table failures are logged as warnings but do not abort the registration of remaining tables.

Returns a list of (table_name, schema) for successfully registered tables.