datafusion-remote-table

Goals

1. Execute SQL queries on remote databases and make results as datafusion table provider
2. Execution plan can be serialized for distributed execution
3. Record batches can be transformed before outputting to next operator

Usage

#[tokio::main]
pub async fn main() {
    let options = ConnectionOptions::Postgres(PostgresConnectionOptions::new(
        "localhost",
        5432,
        "user",
        "password",
    ));
    let remote_table = RemoteTable::try_new(options, "SELECT * from supported_data_types", None, None)
        .await
        .unwrap();

    let ctx = SessionContext::new();
    ctx.register_table("remote_table", Arc::new(remote_table))
        .unwrap();

    ctx.sql("SELECT * from remote_table")
        .await
        .unwrap()
        .show()
        .await
        .unwrap();
}

Supported databases

• Postgres
  • Int2 / Int4 / Int8
  • Float4 / Float8 / Numeric
  • Char / Varchar / Text / Bpchar / Bytea
  • Date / Time / Timestamp / Timestamptz / Interval
  • Bool / Json / Jsonb / Geometry(PostGIS)
  • Int2[] / Int4[] / Int8[]
  • Float4[] / Float8[]
  • Char[] / Varchar[] / Bpchar[] / Text[] / Bytea[]
• MySQL
  • TinyInt (Unsigned) / Smallint (Unsigned) / MediumInt (Unsigned) / Int (Unsigned) / Bigint (Unsigned)
  • Float / Double / Decimal
  • Date / DateTime / Time / Timestamp / Year
  • Char / Varchar / Binary / Varbinary
  • TinyText / Text / MediumText / LongText
  • TinyBlob / Blob / MediumBlob / LongBlob
  • Json / Geometry
• Oracle
  • Varchar2 / Char / Number / Date / Timestamp
• SQLite
  • Null / Integer / Real / Text / Blob

Thanks

• datafusion-table-providers