Crate arrow_odbc

Fill Apache Arrow arrays from ODBC data sources.

Usage

use arrow_odbc::{odbc_api::{Environment, ConnectionOptions}, OdbcReaderBuilder};

const CONNECTION_STRING: &str = "\
    Driver={ODBC Driver 18 for SQL Server};\
    Server=localhost;\
    UID=SA;\
    PWD=My@Test@Password1;\
";

fn main() -> Result<(), anyhow::Error> {
    // Your application is fine if you spin up only one Environment.
    let odbc_environment = Environment::new()?;
     
    // Connect with database.
    let connection = odbc_environment.connect_with_connection_string(
        CONNECTION_STRING,
        ConnectionOptions::default()
    )?;

    // This SQL statement does not require any arguments.
    let parameters = ();

    // Do not apply any timeout.
    let timeout_sec = None;

    // Execute query and create result set
    let cursor = connection
        .execute("SELECT * FROM MyTable", parameters, timeout_sec)?
        .expect("SELECT statement must produce a cursor");

    // Read result set as arrow batches. Infer Arrow types automatically using the meta
    // information of `cursor`.
    let arrow_record_batches = OdbcReaderBuilder::new().build(cursor)?;

    for batch in arrow_record_batches {
        // ... process batch ...
    }

    Ok(())
}

Re-exports

pub use arrow;

pub use odbc_api;

Structs

BufferAllocationOptions

Allows setting limits for buffers bound to the ODBC data source. Check this out if you find that you get memory allocation, or zero sized column errors. Used than constructing a reader using crate::OdbcReaderBuilder.

ConcurrentOdbcReader

Arrow ODBC reader. Implements the arrow::record_batch::RecordBatchReader trait so it can be used to fill Arrow arrays from an ODBC data source. Similar to crate::OdbcReader, yet ConcurrentOdbcReader fetches ODBC batches in a second transit buffer eagerly from the database in a dedicated system thread. This allows the allocation of the Arrow arrays and your application logic to run on the main thread, while fetching the batches from the source happens concurrently. You need twice the memory for the transit buffer for this strategy, since one is may be in use by the main thread in order to copy values into arrow arrays, while the other is used to write values from the database.

OdbcReader

Arrow ODBC reader. Implements the arrow::record_batch::RecordBatchReader trait so it can be used to fill Arrow arrays from an ODBC data source.

OdbcReaderBuilder

Creates instances of OdbcReader based on odbc_api::Cursor.

OdbcWriter

Inserts batches from an arrow::record_batch::RecordBatchReader into a database.

Enums

ColumnFailure

Read error related to a specific column

Error

A variation of things which can go wrong then creating an crate::OdbcReader.

TextEncoding

Used to indicate the preferred encoding for text columns.

WriterError

Emitted writing values from arror arrays into a table on the database

Functions

arrow_schema_from

Query the metadata to create an arrow schema. This method is invoked automatically for you by crate::OdbcReaderBuilder::build. You may want to call this method in situtation ther you want to create an arrow schema without creating the reader yet.

insert_into_table

Fastest and most convinient way to stream the contents of arrow record batches into a database table. For usecase there you want to insert repeatedly into the same table from different streams it is more efficient to create an instance of self::OdbcWriter and reuse it.

insert_statement_from_schema

Creates an SQL insert statement from an arrow schema. The resulting statement will have one placeholer (?) for each column in the statement.