hdbconnect-arrow

Apache Arrow integration for the hdbconnect SAP HANA driver, enabling zero-copy data transfer to analytics tools like Polars and pandas.

Installation

Add to your Cargo.toml:

[dependencies]
hdbconnect-arrow = "0.1"

Or with cargo-add:

cargo add hdbconnect-arrow

[!IMPORTANT] Requires Rust 1.88 or later.

Usage

Basic batch processing

use hdbconnect_arrow::{HanaBatchProcessor, BatchConfig, Result};
use arrow_schema::{Schema, Field, DataType};
use std::sync::Arc;

fn process_results(result_set: hdbconnect::ResultSet) -> Result<()> {
    let schema = Arc::new(Schema::new(vec![
        Field::new("id", DataType::Int64, false),
        Field::new("name", DataType::Utf8, true),
    ]));

    let config = BatchConfig::default();
    let mut processor = HanaBatchProcessor::new(Arc::clone(&schema), config);

    for row in result_set {
        if let Some(batch) = processor.process_row(&row?)? {
            println!("Batch with {} rows", batch.num_rows());
        }
    }

    // Flush remaining rows
    if let Some(batch) = processor.flush()? {
        println!("Final batch with {} rows", batch.num_rows());
    }

    Ok(())
}

Schema mapping

use hdbconnect_arrow::{hana_type_to_arrow, hana_field_to_arrow};
use hdbconnect::TypeId;

// Convert individual types
let arrow_type = hana_type_to_arrow(TypeId::DECIMAL, Some(18), Some(2));
// Returns: DataType::Decimal128(18, 2)

// Convert entire field metadata
let arrow_field = hana_field_to_arrow(&hana_field_metadata);

Custom batch size

use hdbconnect_arrow::BatchConfig;
use std::num::NonZeroUsize;

let config = BatchConfig::new(NonZeroUsize::new(10_000).unwrap());

Features

Enable optional features in Cargo.toml:

[dependencies]
hdbconnect-arrow = { version = "0.1", features = ["async", "test-utils"] }

[!TIP] Enable test-utils in dev-dependencies for unit testing without a HANA connection.

Type mapping

API overview

Core types

• HanaBatchProcessor — Converts HANA rows to Arrow RecordBatch with configurable batch sizes
• BatchConfig — Configuration for batch processing (uses NonZeroUsize for type-safe batch size)
• SchemaMapper — Maps HANA result set metadata to Arrow schemas
• BuilderFactory — Creates appropriate Arrow array builders for HANA types
• TypeCategory — Centralized HANA type classification enum

Traits

• HanaCompatibleBuilder — Trait for Arrow builders that accept HANA values
• FromHanaValue — Sealed trait for type-safe value conversion
• BatchProcessor — Core batch processing interface
• LendingBatchIterator — GAT-based streaming iterator for large result sets
• RowLike — Row abstraction for testing without HANA connection

Test utilities

When test-utils feature is enabled:

use hdbconnect_arrow::{MockRow, MockRowBuilder};

let row = MockRowBuilder::new()
    .push_i64(42)
    .push_string("test")
    .push_null()
    .build();

// Use with HanaBatchProcessor for unit tests

Error handling

use hdbconnect_arrow::{ArrowConversionError, Result};

fn convert_data() -> Result<()> {
    // ArrowConversionError covers:
    // - Type mismatches
    // - Decimal overflow
    // - Schema incompatibilities
    // - Invalid batch configuration
    Ok(())
}

Part of pyhdb-rs

This crate is part of the pyhdb-rs workspace, providing the Arrow integration layer for the Python SAP HANA driver.

Related crates:

• hdbconnect-py — PyO3 bindings exposing Arrow data to Python

MSRV policy

[!NOTE] Minimum Supported Rust Version: 1.88. MSRV increases are minor version bumps.

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.