Struct arrow_odbc::OdbcReader

pub struct OdbcReader<C: Cursor> { /* private fields */ }

Arrow ODBC reader. Implements the arrow::record_batch::RecordBatchReader trait so it can be used to fill Arrow arrays from an ODBC data source.

This reader is generic over the cursor type so it can be used in cases there the cursor only borrows a statement handle (most likely the case then using prepared queries), or owned statement handles (recommened then using one shot queries, to have an easier life with the borrow checker).

Example

use arrow_odbc::{odbc_api::Environment, OdbcReader};

const CONNECTION_STRING: &str = "\
    Driver={ODBC Driver 17 for SQL Server};\
    Server=localhost;\
    UID=SA;\
    PWD=My@Test@Password1;\
";

fn main() -> Result<(), anyhow::Error> {

    let odbc_environment = Environment::new()?;
     
    // Connect with database.
    let connection = odbc_environment.connect_with_connection_string(CONNECTION_STRING)?;

    // This SQL statement does not require any arguments.
    let parameters = ();

    // Execute query and create result set
    let cursor = connection
        .execute("SELECT * FROM MyTable", parameters)?
        .expect("SELECT statement must produce a cursor");

    // Each batch shall only consist of maximum 10.000 rows.
    let max_batch_size = 10_000;

    // Read result set as arrow batches. Infer Arrow types automatically using the meta
    // information of `cursor`.
    let arrow_record_batches = OdbcReader::new(cursor, max_batch_size)?;

    for batch in arrow_record_batches {
        // ... process batch ...
    }
    Ok(())
}

Implementations

impl<C: Cursor> OdbcReader<C>

pub fn new(cursor: C, max_batch_size: usize) -> Result<Self, Error>

Construct a new OdbcReader instance. This constructor infers the Arrow schema from the metadata of the cursor. If you want to set it explicitly use Self::with_arrow_schema.

Parameters

• cursor: ODBC cursor used to fetch batches from the data source. The constructor will bind buffers to this cursor in order to perform bulk fetches from the source. This is usually faster than fetching results row by row as it saves roundtrips to the database. The type of these buffers will be inferred from the arrow schema. Not every arrow type is supported though.
• max_batch_size: Maximum batch size requested from the datasource.

pub fn with_arrow_schema(
    cursor: C,
    max_batch_size: usize,
    schema: SchemaRef
) -> Result<Self, Error>

Construct a new `OdbcReader instance.

Parameters

• cursor: ODBC cursor used to fetch batches from the data source. The constructor will bind buffers to this cursor in order to perform bulk fetches from the source. This is usually faster than fetching results row by row as it saves roundtrips to the database. The type of these buffers will be inferred from the arrow schema. Not every arrow type is supported though.
• max_batch_size: Maximum batch size requested from the datasource.
• schema: Arrow schema. Describes the type of the Arrow Arrays in the record batches, but is also used to determine CData type requested from the data source.

pub fn with(
    cursor: C,
    max_batch_size: usize,
    schema: Option<SchemaRef>,
    buffer_allocation_options: BufferAllocationOptions
) -> Result<Self, Error>

Construct a new crate::OdbcReader instance. This method allows you full control over what options to explicitly specify, and what options you want to leave to this crate to automatically decide.

Parameters

• cursor: ODBC cursor used to fetch batches from the data source. The constructor will bind buffers to this cursor in order to perform bulk fetches from the source. This is usually faster than fetching results row by row as it saves roundtrips to the database. The type of these buffers will be inferred from the arrow schema. Not every arrow type is supported though.
• max_batch_size: Maximum batch size requested from the datasource.
• schema: Arrow schema. Describes the type of the Arrow Arrays in the record batches, but is also used to determine CData type requested from the data source. Set to None to infer schema from the data source.
• buffer_allocation_options: Allows you to specify upper limits for binary and / or text buffer types. This is useful support fetching data from e.g. VARCHAR(max) or VARBINARY(max) columns, which otherwise might lead to errors, due to the ODBC driver having a hard time specifying a good upper bound for the largest possible expected value.

Trait Implementations

impl<C> Iterator for OdbcReader<C>where
    C: Cursor,

type Item = Result<RecordBatch, ArrowError>

The type of the elements being iterated over.

fn next(&mut self) -> Option<Self::Item>

Advances the iterator and returns the next value.

fn next_chunk<const N: usize>(
    &mut self
) -> Result<[Self::Item; N], IntoIter<Self::Item, N>>

🔬This is a nightly-only experimental API. (iter_next_chunk)

Advances the iterator and returns an array containing the next N values.

fn size_hint(&self) -> (usize, Option<usize>)

Returns the bounds on the remaining length of the iterator.

fn count(self) -> usize

Consumes the iterator, counting the number of iterations and returning it.

fn last(self) -> Option<Self::Item>

Consumes the iterator, returning the last element.

fn advance_by(&mut self, n: usize) -> Result<(), usize>

🔬This is a nightly-only experimental API. (iter_advance_by)

Advances the iterator by n elements.

fn nth(&mut self, n: usize) -> Option<Self::Item>

Returns the nth element of the iterator.

fn step_by(self, step: usize) -> StepBy<Self>

Creates an iterator starting at the same point, but stepping by the given amount at each iteration.

fn chain<U>(self, other: U) -> Chain<Self, <U as IntoIterator>::IntoIter>where
    U: IntoIterator<Item = Self::Item>,

Takes two iterators and creates a new iterator over both in sequence.

fn zip<U>(self, other: U) -> Zip<Self, <U as IntoIterator>::IntoIter>where
    U: IntoIterator,

‘Zips up’ two iterators into a single iterator of pairs.

fn intersperse_with<G>(self, separator: G) -> IntersperseWith<Self, G>where
    G: FnMut() -> Self::Item,

🔬This is a nightly-only experimental API. (iter_intersperse)

Creates a new iterator which places an item generated by separator between adjacent items of the original iterator.

fn map<B, F>(self, f: F) -> Map<Self, F>where
    F: FnMut(Self::Item) -> B,

Takes a closure and creates an iterator which calls that closure on each element.

fn for_each<F>(self, f: F)where
    F: FnMut(Self::Item),

Calls a closure on each element of an iterator.

fn filter<P>(self, predicate: P) -> Filter<Self, P>where
    P: FnMut(&Self::Item) -> bool,

Creates an iterator which uses a closure to determine if an element should be yielded.

fn filter_map<B, F>(self, f: F) -> FilterMap<Self, F>where
    F: FnMut(Self::Item) -> Option<B>,

Creates an iterator that both filters and maps.

fn enumerate(self) -> Enumerate<Self>

Creates an iterator which gives the current iteration count as well as the next value.

fn peekable(self) -> Peekable<Self>

Creates an iterator which can use the peek and peek_mut methods to look at the next element of the iterator without consuming it. See their documentation for more information.

fn skip_while<P>(self, predicate: P) -> SkipWhile<Self, P>where
    P: FnMut(&Self::Item) -> bool,

Creates an iterator that skips elements based on a predicate.

fn take_while<P>(self, predicate: P) -> TakeWhile<Self, P>where
    P: FnMut(&Self::Item) -> bool,

Creates an iterator that yields elements based on a predicate.

fn map_while<B, P>(self, predicate: P) -> MapWhile<Self, P>where
    P: FnMut(Self::Item) -> Option<B>,

Creates an iterator that both yields elements based on a predicate and maps.

fn skip(self, n: usize) -> Skip<Self>

Creates an iterator that skips the first n elements.

fn take(self, n: usize) -> Take<Self>

Creates an iterator that yields the first n elements, or fewer if the underlying iterator ends sooner.

fn scan<St, B, F>(self, initial_state: St, f: F) -> Scan<Self, St, F>where
    F: FnMut(&mut St, Self::Item) -> Option<B>,

An iterator adapter similar to fold that holds internal state and produces a new iterator.

fn flat_map<U, F>(self, f: F) -> FlatMap<Self, U, F>where
    U: IntoIterator,
    F: FnMut(Self::Item) -> U,

Creates an iterator that works like map, but flattens nested structure.

fn fuse(self) -> Fuse<Self>

Creates an iterator which ends after the first None.

fn inspect<F>(self, f: F) -> Inspect<Self, F>where
    F: FnMut(&Self::Item),

Does something with each element of an iterator, passing the value on.

fn by_ref(&mut self) -> &mut Self

Borrows an iterator, rather than consuming it.

fn collect<B>(self) -> Bwhere
    B: FromIterator<Self::Item>,

Transforms an iterator into a collection.

fn collect_into<E>(self, collection: &mut E) -> &mut Ewhere
    E: Extend<Self::Item>,

🔬This is a nightly-only experimental API. (iter_collect_into)

Collects all the items from an iterator into a collection.

fn partition<B, F>(self, f: F) -> (B, B)where
    B: Default + Extend<Self::Item>,
    F: FnMut(&Self::Item) -> bool,

Consumes an iterator, creating two collections from it.

fn is_partitioned<P>(self, predicate: P) -> boolwhere
    P: FnMut(Self::Item) -> bool,

🔬This is a nightly-only experimental API. (iter_is_partitioned)

Checks if the elements of this iterator are partitioned according to the given predicate, such that all those that return true precede all those that return false.

fn try_fold<B, F, R>(&mut self, init: B, f: F) -> Rwhere
    F: FnMut(B, Self::Item) -> R,
    R: Try<Output = B>,

An iterator method that applies a function as long as it returns successfully, producing a single, final value.

fn try_for_each<F, R>(&mut self, f: F) -> Rwhere
    F: FnMut(Self::Item) -> R,
    R: Try<Output = ()>,

An iterator method that applies a fallible function to each item in the iterator, stopping at the first error and returning that error.

fn fold<B, F>(self, init: B, f: F) -> Bwhere
    F: FnMut(B, Self::Item) -> B,

Folds every element into an accumulator by applying an operation, returning the final result.

fn reduce<F>(self, f: F) -> Option<Self::Item>where
    F: FnMut(Self::Item, Self::Item) -> Self::Item,

Reduces the elements to a single one, by repeatedly applying a reducing operation.

fn try_reduce<F, R>(
    &mut self,
    f: F
) -> <<R as Try>::Residual as Residual<Option<<R as Try>::Output>>>::TryTypewhere
    F: FnMut(Self::Item, Self::Item) -> R,
    R: Try<Output = Self::Item>,
    <R as Try>::Residual: Residual<Option<Self::Item>>,

🔬This is a nightly-only experimental API. (iterator_try_reduce)

Reduces the elements to a single one by repeatedly applying a reducing operation. If the closure returns a failure, the failure is propagated back to the caller immediately.

fn all<F>(&mut self, f: F) -> boolwhere
    F: FnMut(Self::Item) -> bool,

Tests if every element of the iterator matches a predicate.

fn any<F>(&mut self, f: F) -> boolwhere
    F: FnMut(Self::Item) -> bool,

Tests if any element of the iterator matches a predicate.

fn find<P>(&mut self, predicate: P) -> Option<Self::Item>where
    P: FnMut(&Self::Item) -> bool,

Searches for an element of an iterator that satisfies a predicate.

fn find_map<B, F>(&mut self, f: F) -> Option<B>where
    F: FnMut(Self::Item) -> Option<B>,

Applies function to the elements of iterator and returns the first non-none result.

fn try_find<F, R>(
    &mut self,
    f: F
) -> <<R as Try>::Residual as Residual<Option<Self::Item>>>::TryTypewhere
    F: FnMut(&Self::Item) -> R,
    R: Try<Output = bool>,
    <R as Try>::Residual: Residual<Option<Self::Item>>,

🔬This is a nightly-only experimental API. (try_find)

Applies function to the elements of iterator and returns the first true result or the first error.

fn position<P>(&mut self, predicate: P) -> Option<usize>where
    P: FnMut(Self::Item) -> bool,

Searches for an element in an iterator, returning its index.

fn max_by_key<B, F>(self, f: F) -> Option<Self::Item>where
    B: Ord,
    F: FnMut(&Self::Item) -> B,

Returns the element that gives the maximum value from the specified function.

fn max_by<F>(self, compare: F) -> Option<Self::Item>where
    F: FnMut(&Self::Item, &Self::Item) -> Ordering,

Returns the element that gives the maximum value with respect to the specified comparison function.

fn min_by_key<B, F>(self, f: F) -> Option<Self::Item>where
    B: Ord,
    F: FnMut(&Self::Item) -> B,

Returns the element that gives the minimum value from the specified function.

fn min_by<F>(self, compare: F) -> Option<Self::Item>where
    F: FnMut(&Self::Item, &Self::Item) -> Ordering,

Returns the element that gives the minimum value with respect to the specified comparison function.

fn unzip<A, B, FromA, FromB>(self) -> (FromA, FromB)where
    FromA: Default + Extend<A>,
    FromB: Default + Extend<B>,
    Self: Iterator<Item = (A, B)>,

Converts an iterator of pairs into a pair of containers.

fn copied<'a, T>(self) -> Copied<Self>where
    T: 'a + Copy,
    Self: Iterator<Item = &'a T>,

Creates an iterator which copies all of its elements.

fn cloned<'a, T>(self) -> Cloned<Self>where
    T: 'a + Clone,
    Self: Iterator<Item = &'a T>,

Creates an iterator which clones all of its elements.

fn array_chunks<const N: usize>(self) -> ArrayChunks<Self, N>

🔬This is a nightly-only experimental API. (iter_array_chunks)

Returns an iterator over N elements of the iterator at a time.

fn sum<S>(self) -> Swhere
    S: Sum<Self::Item>,

Sums the elements of an iterator.

fn product<P>(self) -> Pwhere
    P: Product<Self::Item>,

Iterates over the entire iterator, multiplying all the elements

fn cmp_by<I, F>(self, other: I, cmp: F) -> Orderingwhere
    I: IntoIterator,
    F: FnMut(Self::Item, <I as IntoIterator>::Item) -> Ordering,

🔬This is a nightly-only experimental API. (iter_order_by)

Lexicographically compares the elements of this Iterator with those of another with respect to the specified comparison function.

fn partial_cmp<I>(self, other: I) -> Option<Ordering>where
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>,

Lexicographically compares the elements of this Iterator with those of another.

fn partial_cmp_by<I, F>(self, other: I, partial_cmp: F) -> Option<Ordering>where
    I: IntoIterator,
    F: FnMut(Self::Item, <I as IntoIterator>::Item) -> Option<Ordering>,

🔬This is a nightly-only experimental API. (iter_order_by)

Lexicographically compares the elements of this Iterator with those of another with respect to the specified comparison function.

fn eq<I>(self, other: I) -> boolwhere
    I: IntoIterator,
    Self::Item: PartialEq<<I as IntoIterator>::Item>,

Determines if the elements of this Iterator are equal to those of another.

fn eq_by<I, F>(self, other: I, eq: F) -> boolwhere
    I: IntoIterator,
    F: FnMut(Self::Item, <I as IntoIterator>::Item) -> bool,

🔬This is a nightly-only experimental API. (iter_order_by)

Determines if the elements of this Iterator are equal to those of another with respect to the specified equality function.

fn ne<I>(self, other: I) -> boolwhere
    I: IntoIterator,
    Self::Item: PartialEq<<I as IntoIterator>::Item>,

Determines if the elements of this Iterator are unequal to those of another.

fn lt<I>(self, other: I) -> boolwhere
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>,

Determines if the elements of this Iterator are lexicographically less than those of another.

fn le<I>(self, other: I) -> boolwhere
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>,

Determines if the elements of this Iterator are lexicographically less or equal to those of another.

fn gt<I>(self, other: I) -> boolwhere
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>,

Determines if the elements of this Iterator are lexicographically greater than those of another.

fn ge<I>(self, other: I) -> boolwhere
    I: IntoIterator,
    Self::Item: PartialOrd<<I as IntoIterator>::Item>,

Determines if the elements of this Iterator are lexicographically greater than or equal to those of another.

fn is_sorted_by<F>(self, compare: F) -> boolwhere
    F: FnMut(&Self::Item, &Self::Item) -> Option<Ordering>,

🔬This is a nightly-only experimental API. (is_sorted)

Checks if the elements of this iterator are sorted using the given comparator function.

fn is_sorted_by_key<F, K>(self, f: F) -> boolwhere
    F: FnMut(Self::Item) -> K,
    K: PartialOrd<K>,

🔬This is a nightly-only experimental API. (is_sorted)

Checks if the elements of this iterator are sorted using the given key extraction function.

impl<C> RecordBatchReader for OdbcReader<C>where
    C: Cursor,

fn schema(&self) -> SchemaRef

Returns the schema of this RecordBatchReader.

fn next_batch(&mut self) -> Result<Option<RecordBatch>, ArrowError>

👎Deprecated since 2.0.0:

This method is deprecated in favour of next from the trait Iterator.

Reads the next RecordBatch.