Crate nativeclick

Expand description

§Nativeclick

Fork of Protryon/klickhouse (master branch).

Nativeclick is a pure Rust client for ClickHouse over the native TCP protocol, for async (tokio) applications, with derive-based rows and minimal boilerplate.

§Versioning and compatibility

Version scheme: the crate version follows the ClickHouse version it targets, 0.MAJOR.MINOR. 0.26.9 targets ClickHouse 26.9.
Supported servers: each release supports the targeted version and the 10 releases before it. For 0.26.9 that is ClickHouse 25.11 to 26.9, and CI runs the whole test suite against each of them.
Protocol: the client speaks native protocol revision 54493, and each connection negotiates min(client, server). Every revision-dependent field and column format follows the negotiated revision, so older servers keep working.
Rust: MSRV is 1.89, tested in CI.

See CHANGELOG.md for the changes since klickhouse and how to migrate.

§nativeclick vs klickhouse

Compared with klickhouse 0.15 (the fork point, protocol revision 54448):

	klickhouse 0.15	nativeclick 0.26.9
Compatibility
Protocol revision	54448 (ClickHouse 22.x era)	54493, negotiated per connection
ClickHouse 26.9 (ZSTD answers by default)	❌ empty results, then the connection dies	✅
Versions tested in CI	25.8	25.11 → 26.9 (11 versions)
Compressed answers read	LZ4 only	LZ4, ZSTD, none
Sparse / replicated columns, size-stream strings	❌ (revision too old to receive them)	✅
Reliability
Unsupported column type	❌ “OK, 0 rows”, then the connection silently dies	✅ clear error naming the type
Concurrent use of a cloned `Client`	❌ can corrupt the stream	✅
INSERT row that fails to serialize	❌ silently skipped	✅ error, nothing sent for that block
Server-side INSERT errors (constraints…)	❌ lost	✅ returned
`DateTime64` before 1970	❌ panics	✅
`Decimal(P, 0)`, `DateTime64(0)` inserts	❌ rejected	✅
Server log packets (`send_logs_level`)	❌ panic the connection	✅
Malformed server data	panics (`unimplemented!`, asserts)	errors for offsets, maps, kinds, packets
Features
Per-query settings	❌	✅ `QueryOptions::setting`
Server-side `{name:Type}` parameters	❌	✅ `param`, `param_string`
Query id	random, hidden	✅ settable, used in progress events
Query timeout	❌	✅ cancels on the server
Cancel a running query	❌ keeps running	✅ on stream drop or timeout
Connect timeout, quota key	❌	✅ `ClientOptions`
Ping, server info (version, timezone, profile settings)	❌	✅ `ping()`, `server_info()`
TLS in the bb8 pool	❌	✅ `ConnectionManager::with_tls`
Types
`Bool`	as `UInt8`	✅ own type (`u8` still reads it)
`Date32`, `Time`, `Time64`, `Interval*`	❌	✅
`Nothing` (`SELECT NULL`, `[]`)	❌	✅
Named tuples	❌	✅
`Enum` names with `,` `'` `(`	❌ misparsed	✅, read/write by value or by name
`LineString`, `MultiLineString`, `MultiPoint`, `Geometry`	❌	✅
`Variant`, `Dynamic`, `JSON`	❌	✅
`SimpleAggregateFunction`	❌	✅
`AggregateFunction`	❌	❌ by design, explicit error pointing to `finalizeAggregation`
Code health
Unmaintained / dead dependencies	`paste`, `compiler-tools`, `rustc_version` + dead `build.rs`	removed
Error kinds	catch-all `ProtocolError(String)`	`ConnectionClosed`, `Timeout`, `Compression`, `Unsupported`, …
Tests	13 integration tests, run on one version	116 unit + 61 integration, on 11 versions

Decoding speed of this crate’s decoder before and after the 0.26.9 optimizations. Measured with examples/throughput.rs (2M rows × 5 columns, ClickHouse 26.9 on localhost, ZSTD answers):

	before	after
SELECT, blocks	4.0 M rows/s	6.1 M rows/s
SELECT, derived rows	2.8 M rows/s	3.6 M rows/s
INSERT, derived rows	0.94 M rows/s	1.5 M rows/s

§Getting started

[dependencies]
nativeclick = "0.26.9"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

use nativeclick::{Client, ClientOptions, Row};

#[derive(Row, Debug, Clone)]
struct Event {
    id: u64,
    name: String,
    tags: Vec<String>,
    score: Option<f64>,
}

#[tokio::main]
async fn main() -> nativeclick::Result<()> {
    let client = Client::connect("127.0.0.1:9000", ClientOptions::default()).await?;

    client
        .execute("CREATE TABLE IF NOT EXISTS events (id UInt64, name String, tags Array(String), score Nullable(Float64)) ENGINE = MergeTree ORDER BY id")
        .await?;

    client
        .insert_native_block(
            "INSERT INTO events FORMAT Native",
            vec![Event { id: 1, name: "start".into(), tags: vec!["a".into()], score: None }],
        )
        .await?;

    let events: Vec<Event> = client.query_collect("SELECT * FROM events").await?;
    println!("{events:?}");
    Ok(())
}

§Queries

Method	Use
`query::<T>`	stream of rows `T: Row`
`query_collect`, `query_one`, `query_opt`	all rows, the first one, or the first one if any
`query_raw`	stream of raw `Block`s
`execute`	run a statement and wait for it; `execute_now` does not wait
`insert_native`, `insert_native_block`	insert rows; the query must end with `FORMAT Native`
`insert_native_raw`	insert raw `Block`s

Rows are any struct with #[derive(Row)], RawRow (dynamic access by index or name), or UnitValue<T> for a single column. Field names match column names, and can be adjusted with #[nativeclick(rename = "...")], rename_all, default, flatten, nested.

§Per-query options

Client::with_options returns a handle on the same connection whose queries use the given options. The original handle is unchanged.

use std::time::Duration;
use nativeclick::{QueryOptions, Uuid};

let id = Uuid::new_v4();
let tuned = client.with_options(
    QueryOptions::new()
        .query_id(id)                          // visible in system.query_log, KILL QUERY
        .setting("max_threads", 4)             // any ClickHouse setting
        .setting("max_execution_time", 60)
        .param("min_id", 10)                   // {min_id:UInt64}
        .param("ids", "[1, 2, 3]")             // {ids:Array(UInt8)}, as clickhouse-client --param_ids
        .param_string("name", "it's \"quoted\"") // any text for a {name:String}, escaped for you
        .timeout(Duration::from_secs(30)),     // cancels the query on the server when exceeded
);
tuned
    .execute("SELECT count() FROM events WHERE id >= {min_id:UInt64} AND name != {name:String}")
    .await?;

Parameters are values sent separately from the SQL, so they cannot inject SQL.
- param takes the text exactly as clickhouse-client --param_x=... does: numbers, dates, [1,2], ['a','b'].
- param_string takes any text for a String parameter and escapes it.
Timeouts fail the query with NativeclickError::Timeout and cancel it on the server, so the connection stays usable.
Dropping a result stream before its end also cancels the query on the server.

§Connection

use std::time::Duration;
use nativeclick::{Client, ClientOptions};

let client = Client::connect(
    "127.0.0.1:9000",
    ClientOptions {
        username: "default".into(),
        password: String::new(),
        default_database: "default".into(),
        quota_key: "tenant-42".into(),
        connect_timeout: Some(Duration::from_secs(5)),
        ..Default::default()
    },
)
.await?;

let server = client.server_info();
println!("{} {}.{} ({:?}), protocol {}", server.name, server.version_major, server.version_minor, server.timezone, server.negotiated_revision);
client.ping().await?;

A Client is cheap to clone and can be used from many tasks; queries on one connection run one after the other. For parallelism, use several connections, for example with the bb8 pool:

// features = ["bb8"] (and "tls" for with_tls)
let manager = nativeclick::ConnectionManager::new("127.0.0.1:9000", ClientOptions::default()).await?;
let pool = nativeclick::bb8::Pool::builder().max_size(8).build(manager).await?;
let client = pool.get().await?;

Connection-level failures are reported to every waiting call, and later calls on the same Client return the error that closed it. Client::is_closed() tells when to reconnect, and the bb8 pool does it for you.

§Errors

NativeclickError is #[non_exhaustive] and Clone. The main variants:

Variant	Meaning
`ServerException { code, name, message, .. }`	ClickHouse rejected the query
`Timeout`	`QueryOptions::timeout` or `ClientOptions::connect_timeout` expired
`ConnectionClosed`	the connection is gone
`DeserializeError`, `SerializeError`, `UnexpectedType*`, `TypeParseError`	a value or type does not match
`Compression`, `ProtocolError`	corrupt frame or unexpected server data
`Unsupported`	the server is too old for a requested feature

§Types

ClickHouse	Rust
`Int8`…`Int256`, `UInt8`…`UInt256`, `Float32/64`, `BFloat16`	integers, floats, `i256`/`u256`, `bf16`
`Bool`	`bool` (`u8` also reads it)
`Decimal(P, S)`, `Decimal32/64/128/256`	`FixedPoint32/64/128/256<S>`, `rust_decimal::Decimal`
`String`, `FixedString(N)`	`String`, `Bytes`; `Vec<u8>` writes
`UUID`, `IPv4`, `IPv6`	`Uuid`, `Ipv4`, `Ipv6`
`Date`, `Date32`	`Date`, `Date32`, `chrono::NaiveDate`
`DateTime`, `DateTime64(p)`	`DateTime`, `DateTime64<P>` (`from_ticks`/`ticks` for dates before 1970), `chrono::DateTime`
`Time`, `Time64(p)`	`chrono::TimeDelta`
`Interval*`	`i64`
`Enum8`, `Enum16`	`i8`/`i16` (value) or `String` (name)
`Nullable(T)`, `Nothing`	`Option<T>`
`Array(T)`	`Vec<T>`, `[T; N]`
`Map(K, V)`	`HashMap`, `BTreeMap`, `IndexMap`
`Tuple(...)`, named tuples	Rust tuples
`LowCardinality(T)`, `SimpleAggregateFunction(f, T)`	as `T`
`Point`, `Ring`, `LineString`, `MultiPoint`, `Polygon`, `MultiLineString`, `MultiPolygon`	`Point`, `Ring`, `Polygon`, `MultiPolygon` (and `geo-types` with the feature)
`Variant(...)`, `Dynamic`, `Geometry`	`Value` / `DynamicValue` (value plus actual type), or `T` / `Option<T>` when rows have that type
`JSON`	`String` (JSON text), `Json<T>` (serde)
`AggregateFunction(...)`	not readable: states have no generic format. Select `finalizeAggregation(col)` instead.

Notes:

JSON and Dynamic formats: the client asks the server for JSON columns as text and for the flattened Dynamic layout, with two settings sent before the caller’s own (which can override them).
Writing Variant / Dynamic / Geometry:
- a DynamicValue chooses the type explicitly;
- a plain value goes to the first variant that accepts it (Variant) or to its natural type (Dynamic).
Recent column formats: sparse and replicated columns, and the size-stream string layout (revision 54492), are decoded transparently.

§Feature flags

derive: the #[derive(Row)] macro. Default.
compression: compression of client/server traffic: writes lz4, reads lz4 and zstd. Default.
serde: serde support on values, and the Json<T> wrapper. Default.
bfloat16: the BFloat16 type. Default.
tls: TLS via tokio-rustls, for Client::connect_tls and the bb8 pool.
bb8: a bb8 ConnectionManager.
refinery: migrations via refinery.
geo-types: conversions with the geo-types crate.
rust_decimal: conversions with rust_decimal.

§Running the tests

The integration tests need a ClickHouse server:

docker run -d --name clickhouse -p 19000:9000 -e CLICKHOUSE_SKIP_USER_SETUP=1 --ulimit nofile=262144:262144 clickhouse/clickhouse-server:26.9
export NATIVECLICK_TEST_ADDR=127.0.0.1:19000
# export NATIVECLICK_TEST_USER=default NATIVECLICK_TEST_PASSWORD= NATIVECLICK_TEST_DATABASE=default
cargo test --all-features -- --test-threads=1

The tests share one server and its tables, so run them on one thread.

To measure throughput:

NATIVECLICK_TEST_ADDR=127.0.0.1:19000 cargo run --release --example throughput

§Credit

Nativeclick started as a fork of klickhouse by Protryon. nativeclick_derive was made by copy/paste/simplify of serde_derive to get maximal functionality and performance at the lowest time cost. In a prototype, serde was used directly, but this was abandoned because of the lock-in of serde’s data model.

Re-exports§

pub use bb8;

Modules§

block
query_parser: Minimal SQL lexing for client-side $N arguments and for splitting statements: it only needs to know where strings, quoted identifiers, comments and heredocs start and end.

Structs§

Bytes: Wrapper over Vec<u8> to allow more efficient serialization/deserialization of raw bytes The corresponding Clickhouse type here is String or FixedString, not Array(UInt8). Conversion to Array(UInt8) will happen, but it is not efficient.
ClickhouseLock: A hack implementation of a global lock for things like migrations
Client: Client handle for a Clickhouse connection, has internal reference to connection, and can be freely cloned and sent across threads.
ClientOptions: Options set for a Clickhouse connection.
ClusterMigration: Wrapper for Client to use migrations on clusters
ConnectionManager: bb8 connection manager for Client: plaintext TCP by default, TLS with ConnectionManager::with_tls.
Date: Wrapper type for Clickhouse Date type.
Date32: Wrapper type for Clickhouse Date32 type: days since 1970-01-01, signed.
DateTime: Wrapper type for Clickhouse DateTime type.
DateTime64: Wrapper type for Clickhouse DateTime64 type: ticks of 10^-PRECISION seconds since the Unix epoch.
DynDateTime64: Wrapper type for Clickhouse DateTime64 type with dynamic precision. Same raw-bits representation as DateTime64.
DynamicValue: A row of a Variant, Dynamic or Geometry column: the value and its actual type.
FixedPoint32: Wrapper type for Clickhouse FixedPoint32 type.
FixedPoint64: Wrapper type for Clickhouse FixedPoint64 type.
FixedPoint128: Wrapper type for Clickhouse FixedPoint128 type.
FixedPoint256: Wrapper type for Clickhouse FixedPoint256 type.
IndexMap: A hash table where the iteration order of the key-value pairs is independent of the hash values of the keys.
Ipv4: Wrapper type for Clickhouse IPv4 type.
Ipv6: Wrapper type for Clickhouse IPv6 type.
Json: A Vec wrapper that is encoded as a tuple in SQL as opposed to a Vec
MultiPolygon: Union of polygons.
ParsedQuery
Point: Geo point, represented by its x and y coordinates.
Polygon: Polygon with holes. The first element is the outer polygon, and the following ones are the holes.
Progress: Query execution progress. Values are delta and must be summed.
QueryBuilder
QueryOptions: Options applied to queries, see Client::with_options.
RawRow: A row of raw data returned from the database by a query. Or an unstructured runtime-defined row to upload to the server.
Ring: Polygon without holes.
SelectBuilder
ServerInfo: What the server told about itself when connecting.
UnitValue: A single column row
Uuid: A Universally Unique Identifier (UUID).
VecTuple: A Vec wrapper that is encoded as a tuple in SQL as opposed to a Vec
bf16: A 16-bit floating point type implementing the bfloat16 format.
i256: Wrapper type for Clickhouse Int256 type.
u256: Wrapper type for Clickhouse UInt256 type.

Enums§

NativeclickError: Every error of this crate. Errors are Clone: one connection error is handed to every caller waiting on that connection.
Type: A raw Clickhouse type.
Tz: TimeZones built at compile time from the tz database
Value: A raw Clickhouse value. Types are not strictly/completely preserved (i.e. types String and FixedString both are value String). Use this if you want dynamically typed queries.

Constants§

MIN_SUPPORTED_VERSION: Oldest ClickHouse version this release is tested against: the targeted version and the 10 releases before it are supported.
VERSION_MAJOR: Clickhouse major version this crate targets, sent to the server as the client version. The crate version is 0.MAJOR.MINOR (0.26.9 targets ClickHouse 26.9).
VERSION_MINOR: Clickhouse minor version this crate targets, sent to the server as the client version.

Traits§

FromSql: A type that can be converted from a raw Clickhouse SQL value.
Row: A row that can be deserialized and serialized from a raw Clickhouse SQL value. Generally this is not implemented manually, but using nativeclick_derive::Row. I.e. #[derive(nativeclick::Row)].
RowIndex
ToSql: A type that can be converted to a raw Clickhouse SQL value.

Functions§

from_sql_resolved: Converts value, read from a column of type type_, into T.
unexpected_type

Type Aliases§

Result

Derive Macros§

Row: Derive macro for the Row trait.