Crate clickhouse_lexer_sys 

clickhouse-lexer-sys

Rust bindings to the actual SQL lexer of ClickHouse (src/Parsers/Lexer.cpp). The sources are copied unchanged from ClickHouse/clickhouse-lexer, the standalone build ClickHouse maintains for its web UI, and compiled natively at build time. TokenKind is generated from the same Lexer.h, so the token numbering can never drift from the compiled lexer.

Requires clang++ (ClickHouse only supports clang; set CXX to point at it if it is not on PATH). There is no C++ runtime dependency.

Besides the raw bindings in ffi, the crate ships a minimal safe wrapper, Lexer, so a separate wrapper crate is not needed for the common case.

Usage

use clickhouse_lexer_sys::{Lexer, TokenKind};

let tokens: Vec<_> = Lexer::new("SELECT count() FROM `events` WHERE user != 'guest' -- filter")
    .filter(|t| t.kind.is_significant())
    .collect();

assert_eq!(tokens[0].kind, TokenKind::BareWord);
assert_eq!(tokens[0].text, b"SELECT");
assert_eq!(tokens.last().unwrap().kind, TokenKind::StringLiteral);

// The lexer stops at unlexable input and reports what went wrong:
let last = Lexer::new("SELECT 'unterminated").last().unwrap();
assert_eq!(last.kind, TokenKind::ErrorSingleQuoteIsNotClosed);

Each Token is {kind, begin, end, text}. Tokens tile the input from the start: every byte up to the end of the last token belongs to exactly one token, whitespace and comments included. begin and end are byte offsets, text is the covered bytes, and text_str() decodes it when it is valid UTF-8. Iteration ends after the first error token, which is yielded, or at the end of the input, where TokenKind::EndOfStream is not yielded; Lexer::next_token exposes the raw stream including EndOfStream.

Item	Description
Lexer::new(input)	Lex a &str or &[u8].
Lexer::with_max_query_size(input, n)	Stop with ErrorMaxQuerySizeExceeded past n bytes.
TokenKind	The upstream TokenType enum, same names, same order.
TokenKind::{is_significant, is_error, is_end}	The upstream predicates.
TokenKind::{from_raw, name, ALL}	Conversions for the raw C API.
UPSTREAM_COMMIT, UPSTREAM_VERSION	The ClickHouse commit and clickhouse-lexer release the sources come from.
ffi	The raw C API from upstream/clickhouse_lexer.h.

TokenKind is #[non_exhaustive]: upstream adds token types from time to time, and a sync then lands in a patch release.

How this repository is synced

upstream/ holds the lexer sources verbatim from the upstream/ directory of a clickhouse-lexer release, which in turn mirrors src/Parsers of ClickHouse. upstream/VERSION names that release and upstream/COMMIT the ClickHouse commit it was taken from.

The Sync workflow runs weekly:

1. scripts/sync.sh copies upstream/ from the latest clickhouse-lexer release and stops if nothing changed.
2. It bumps the patch version, runs cargo test, and opens a pull request.

CI checks every change on Linux and macOS: the Rust tests, clippy, rustfmt and docs, that upstream/ is byte-identical to the recorded release, and that clickhouse-lexer’s own test suite passes when built from the shipped sources.

To release after merging a sync pull request, push the matching tag:

git tag "v$(cargo metadata --no-deps --format-version 1 | jq -r '.packages[0].version')"
git push --tags

The Release workflow checks that the tag matches Cargo.toml and sits on main, runs CI, publishes to crates.io through trusted publishing, and creates the GitHub release. One-time setup: publish the first version by hand with cargo publish, then on the crate’s crates.io settings page add a trusted publisher for this repository with workflow release.yml and environment release.

License

Apache License 2.0, same as ClickHouse. The upstream/ sources are verbatim copies from the ClickHouse repository and remain © ClickHouse, Inc.

Modules

ffi

The C API of upstream/clickhouse_lexer.h, as compiled by this crate.

Structs

Lexer

ClickHouse’s lexer over one input.

Token

A token of the input: its kind and the byte range it covers.

Enums

TokenKind

The TokenType enum of ClickHouse’s src/Parsers/Lexer.h, in the same order.

Constants

UPSTREAM_COMMIT

The ClickHouse/ClickHouse commit the vendored lexer sources come from.

UPSTREAM_VERSION

The ClickHouse/clickhouse-lexer release the vendored sources were copied from.