Crate connector_arrow

A flexible database client that converts data into Apache Arrow format across various databases.

The API provided by each data source is described in api module.

Capabilities:

• Query: Query databases and retrieve results in Apache Arrow format.
• Query Parameters: Utilize Arrow type system for query parameters.
• Temporal and Container Types: Correctly handles temporal and container types.
• Schema Introspection: Query the database for schema of specific tables.
• Schema Migration: Basic schema migration commands.
• Append: Write arrow::record_batch::RecordBatch into database tables.

Example for SQLite:

use connector_arrow::api::{Connector, Statement, ResultReader};
use connector_arrow::arrow;

// a regular rusqlite connection
let conn = rusqlite::Connection::open_in_memory()?;

// wrap into connector_arrow connection
let mut conn = connector_arrow::rusqlite::SQLiteConnection::new(conn);

let mut stmt = conn.query("SELECT 1 as a")?;

let mut reader = stmt.start([])?;

let schema: arrow::datatypes::SchemaRef = reader.get_schema()?;

// reader implements Iterator<Item = Result<RecordBatch, _>>
let batches: Vec<arrow::record_batch::RecordBatch> = reader.collect::<Result<_, _>>()?;

For a list of supported databases, refer to the crates.io page.

Transitive dependency on arrow

If you depend on connector_arrow, it is recommended not to depend on arrow directly, but use re-export from this crate instead. This advice is only relevant if your crate does not need any additional arrow features.

use connector_arrow::arrow;

If you do depend on arrow directly, you have to make sure to use exactly the same version as is used by connector_arrow, otherwise types from arrow and connector_arrow will not be interchangeable and might lead to type errors.

This situation is made much worse by unusually high cadence of major version releases of arrow-rs, even without breaking changes.

Re-exports

pub use arrow;

Modules

api

Database client interface that uses Apache Arrow as data-transfer format and schema definition format.

duckdb

Provides connector_arrow traits for duckdb crate.

mysql

postgres

Provides connector_arrow traits for postgres crate.

rusqlite

Provides connector_arrow traits for rusqlite crate.

tiberius

types

util

Utilities for converting row-major tabular data into Apache Arrow. Used by database client implementations.

Macros

impl_consume_unsupported

impl_produce_unsupported

Enums

ConnectorError

Errors that can be raised from this library.

TableCreateError

TableDropError

Functions

query

Open a connection, execute a single query and return the results.