Crate clickhouse_arrow

Expand description

§🛰️ `ClickHouse` Native Protocol Rust Client w/ Arrow Compatibility

ClickHouse access in rust over ClickHouse’s native protocol.

A high-performance, async Rust client for ClickHouse with native Arrow integration. Designed to be faster and more memory-efficient than existing alternatives.

§Why clickhouse-arrow?

🚀 Performance: Optimized for speed with zero-copy deserialization where possible
🎯 Arrow Native: First-class Apache Arrow support for efficient data interchange
📊 90%+ Test Coverage: Comprehensive test suite ensuring reliability
🔄 Async/Await: Modern async API built on Tokio
🗜️ Compression: LZ4 and ZSTD support for efficient data transfer
☁️ Cloud Ready: Full ClickHouse Cloud compatibility
🛡️ Type Safe: Compile-time type checking with the #[derive(Row)] macro

§Details

The crate supports two “modes” of operation:

§`ArrowFormat`

Support allowing interoperability with arrow.

§`NativeFormat`

Uses internal types and custom traits if a dependency on arrow is not required.

§`CreateOptions`, `SchemaConversions`, and Schemas

§Creating Tables from Arrow Schemas

clickhouse-arrow provides powerful DDL capabilities through CreateOptions, allowing you to create ClickHouse tables directly from Arrow schemas:

use clickhouse_arrow::{Client, ArrowFormat, CreateOptions};
use arrow::datatypes::{Schema, Field, DataType};

// Define your Arrow schema
let schema = Schema::new(vec![
    Field::new("id", DataType::UInt64, false),
    Field::new("name", DataType::Utf8, false),
    Field::new("status", DataType::Dictionary(Box::new(DataType::Int8), Box::new(DataType::Utf8)), false),
]);

// Configure table creation
let options = CreateOptions::new("MergeTree")
    .with_order_by(&["id".to_string()])
    .with_partition_by("toYYYYMM(created_at)")
    .with_setting("index_granularity", 8192);

// Create the table
client.create_table(None, "my_table", &schema, &options, None).await?;

§Schema Conversions for Type Control

SchemaConversions (type alias for HashMap<String, Type>) provides fine-grained control over Arrow-to-ClickHouse type mappings. This is especially important for:

Converting Dictionary → Enum: By default, Arrow Dictionary types map to LowCardinality(String). Use SchemaConversions to map them to Enum8 or Enum16 instead:

use clickhouse_arrow::{Type, CreateOptions};
use std::collections::HashMap;

let schema_conversions = HashMap::from([
    // Convert status column from Dictionary to Enum8
    ("status".to_string(), Type::Enum8(vec![
        ("active".to_string(), 0),
        ("inactive".to_string(), 1),
        ("pending".to_string(), 2),
    ])),
    // Convert category to Enum16 for larger enums
    ("category".to_string(), Type::Enum16(vec![
        ("electronics".to_string(), 0),
        ("clothing".to_string(), 1),
        // ... up to 65k values
    ])),
]);

let options = CreateOptions::new("MergeTree")
    .with_order_by(&["id".to_string()])
    .with_schema_conversions(schema_conversions);

Geo Types: Preserve geographic types during conversion
Date Types: Choose between Date and Date32
Custom Type Mappings: Override any default type conversion

§Field Naming Constants

When working with complex Arrow types, use these constants to ensure compatibility:

use clickhouse_arrow::arrow::types::*;

// For List types - inner field is named "item"
let list_field = Field::new("data", DataType::List(
    Arc::new(Field::new(LIST_ITEM_FIELD_NAME, DataType::Int32, true))
), true);

// For Struct/Tuple types - fields are named "field_0", "field_1", etc.
let tuple_fields = vec![
    Field::new(format!("{}{}", TUPLE_FIELD_NAME_PREFIX, 0), DataType::Int32, false),
    Field::new(format!("{}{}", TUPLE_FIELD_NAME_PREFIX, 1), DataType::Utf8, false),
];

// For Map types - uses specific field names
let map_type = DataType::Map(
    Arc::new(Field::new(MAP_FIELD_NAME, DataType::Struct(
        vec![
            Field::new(STRUCT_KEY_FIELD_NAME, DataType::Utf8, false),
            Field::new(STRUCT_VALUE_FIELD_NAME, DataType::Int32, true),
        ].into()
    ), false)),
    false
);

These constants ensure your Arrow schemas align with ClickHouse’s expectations and maintain compatibility with arrow-rs conventions.

§Queries

§Query Settings

The clickhouse_arrow::Settings type allows configuring ClickHouse query settings. You can import it directly:

use clickhouse_arrow::Settings;
// or via prelude
use clickhouse_arrow::prelude::*;

Refer to the settings module documentation for details and examples.

§Arrow Round-Trip

There are cases where a round trip may deserialize a different type by schema or array than the schema and array you used to create the table.

will try to maintain an accurate and updated list as they occur. In addition, when possible, I will provide options or other functionality to alter this behavior.

§`(String|Binary)View`/`Large(List|String|Binary)` variations are normalized.

Behavior: ClickHouse does not make the same distinction between Utf8, Utf8View, or LargeUtf8. All of these are mapped to either Type::Binary (the default, see above) or Type::String
Option: None
Default: Unsupported
Impact: When deserializing from ClickHouse, manual modification will be necessary to use these data types.

§`Utf8` -> `Binary`

Behavior: By default, Type::String/DataType::Utf8 will be represented as Binary.
Option: strings_as_strings (default: false).
Default: Disabled (false).
Impact: Set to true to strip map Type::String -> DataType::Utf8. Binary tends to be more efficient to work with in high throughput scenarios

§Nullable `Array`s

Behavior: ClickHouse does not allow Nullable(Array(...)), but insertion with non-null data is allowed by default. To modify this behavior, set array_nullable_error to true.
Option: array_nullable_error (default: false).
Default: Disabled (false).
Impact: Enables flexible insertion but may cause schema mismatches if nulls are present.

§`LowCardinality(Nullable(...))` vs `Nullable(LowCardinality(...))`

Behavior: Like arrays mentioned above, ClickHouse does not allow nullable low cardinality. The default behavior is to push down the nullability.
Option: low_cardinality_nullable_error (default: false).
Default: Disabled (false).
Impact: Enables flexible insertion but may cause schema mismatches if nulls are present.

§`Enum8`/`Enum16` vs. `LowCardinality`

Behavior: Arrow Dictionary types map to LowCardinality, but ClickHouse Enum types may also map to Dictionary, altering the type on round-trip.
Option: No options available rather provide hash maps for either enum_i8 and/or enum_i16 for CreateOptions during schema creation.
Impact: The default behavior will ignore enums when starting from arrow.

Re-exports§

pub use native::progress::Progress;
pub use native::CompressionMethod;
pub use native::ServerError;
pub use native::Severity;
pub use bb8;
pub use rustc_hash;
pub use tracing;
pub use native::convert::*;
pub use native::types::*;
pub use native::values::*;

Modules§

arrow: Logic for interfacing between Arrow and ClickHouse
native: Logic for interfacing between internal ‘native’ types and ClickHouse
prelude: Convenience exports for working with the library.
spawn: A wrapper around a JoinHandle that handles panics.
telemetry: Tracing (telemetry) utilities and constants.

Structs§

ArrowFormat: Marker trait for Arrow format.
ArrowOptions: Configuration options for Arrow serialization and deserialization with ClickHouse.
ClickHouseResponse
Client: A thread-safe handle for interacting with a ClickHouse database over its native protocol.
ClientBuilder: A builder for configuring and creating a ClickHouse client.
ClientOptions: Configuration options for a ClickHouse client connection and Arrow serialization.
CloudOptions: Configuration options for connecting to ClickHouse cloud instances.
ConnectionContext: Configuration for a ClickHouse connection, including tracing and cloud-specific settings.
ConnectionManager: ConnectionManager is the underlying manager that bb8::Pool uses to manage connections.
ConnectionPoolBuilder: Helper to construct a bb8 connection pool
CreateOptions: Options for creating a ClickHouse table, specifying engine, ordering, partitioning, and other settings.
Destination
Event: Emitted clickhouse events from the underlying connection
ExponentialBackoff
Extension: Extra configuration options for ClickHouse.
IndexMap: A hash table where the iteration order of the key-value pairs is independent of the hash values of the keys.
NativeFormat: Marker for Native format.
ParsedQuery: Represents a parsed query.
ProfileEvent: Emitted by ClickHouse during operations.
Qid: An internal representation of a query id, meant to reduce costs when tracing, passing around, and converting to strings.
QueryParams: Represent parameters that can be passed to bind values during queries.
Setting: A single ClickHouse query setting, consisting of a key, value, and flags.
Settings: A collection of ClickHouse query settings.
Uuid: A Universally Unique Identifier (UUID).

Enums§

ChunkedProtocolMode
ClickHouseEvent: Profile and progress events from clickhouse
ConnectionStatus: The status of the underlying connection to ClickHouse
Error: Represents various library errors
SettingValue: Supported value types for ClickHouse query settings.
Tz: TimeZones built at compile time from the tz database

Constants§

CONN_READ_BUFFER_ENV_VAR: Set this environment to enable additional debugs around arrow (de)serialization.
CONN_WRITE_BUFFER_ENV_VAR: Set this environment to enable additional debugs around arrow (de)serialization.
DEBUG_ARROW_ENV_VAR: Set this environment to enable additional debugs around arrow (de)serialization.

Traits§

ClientFormat: Marker trait for various client formats.

Type Aliases§

ArrowClient: A ClickHouse client configured for Apache Arrow format.
ArrowConnectionManager: Alias for ConnectionManager<ArrowFormat>
ArrowConnectionPoolBuilder: Alias for ConnectionPoolBuilder<ArrowFormat>
ArrowPoolBuilder: Alias for bb8::Builder<ConnectionManager<ArrowFormat>>
ConnectionPool: Alias for bb8::Pool<ConnectionManager<T>>
FxIndexMap: A non-cryptographically secure indexmap::IndexMap using HashBuilder.
HashBuilder: A non-cryptographically secure std::hash::BuildHasherDefault using rustc_hash::FxHasher.
NativeClient: A ClickHouse client configured for the native format.
NativeConnectionManager: Alias for ConnectionManager<NativeFormat>
NativeConnectionPoolBuilder: Alias for ConnectionPoolBuilder<NativeFormat>
NativePoolBuilder: Alias for bb8::Builder<ConnectionManager<NativeFormat>>
ParamValue: Type alias to help distinguish settings from params
PoolBuilder: Alias for bb8::Builder<ConnectionManager<T>>
Result

Derive Macros§

Row: Derive macro for the Row trait.