datafusion-remote-table

Features

1. Execute SQL queries on remote databases and stream results as datafusion table provider
2. Support pushing down filters and limit to remote databases
3. Execution plan can be serialized for distributed execution
4. Record batches can be transformed before outputting to next plan node

Usage

#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let options = ConnectionOptions::Postgres(PostgresConnectionOptions::new(
        "localhost",
        5432,
        "user",
        "password",
    ));
    let remote_table = RemoteTable::try_new(options, "SELECT * from supported_data_types").await?;

    let ctx = SessionContext::new();
    ctx.register_table("remote_table", Arc::new(remote_table))?;

    ctx.sql("SELECT * from remote_table").await?.show().await?;

    Ok(())
}

Supported databases

• Postgres
  • Int2 / Int4 / Int8
  • Float4 / Float8 / Numeric
  • Char / Varchar / Text / Bpchar / Bytea
  • Date / Time / Timestamp / Timestamptz / Interval
  • Bool / Oid / Name / Json / Jsonb / Geometry(PostGIS)
  • Int2[] / Int4[] / Int8[]
  • Float4[] / Float8[]
  • Char[] / Varchar[] / Bpchar[] / Text[] / Bytea[]
• MySQL
  • TinyInt (Unsigned) / Smallint (Unsigned) / MediumInt (Unsigned) / Int (Unsigned) / Bigint (Unsigned)
  • Float / Double / Decimal
  • Date / DateTime / Time / Timestamp / Year
  • Char / Varchar / Binary / Varbinary
  • TinyText / Text / MediumText / LongText
  • TinyBlob / Blob / MediumBlob / LongBlob
  • Json / Geometry
• Oracle
  • Number / BinaryFloat / BinaryDouble / Float
  • Varchar2 / NVarchar2 / Char / NChar / Long / Clob / NClob
  • Raw / Long Raw / Blob
  • Date / Timestamp
  • Boolean
• SQLite
  • Null / Integer / Real / Text / Blob

Thanks

• datafusion-table-providers