faucet-source-clickhouse

ClickHouse query source for the faucet-stream ecosystem.

Talks to ClickHouse over its HTTP interface using reqwest: runs a SQL SELECT, requests the JSONEachRow output format, and streams the response body straight into StreamPages. Response bytes are line-buffered and decoded incrementally, so memory stays bounded (batch_size records per page) regardless of how large the result set is.

Configuration

source:
  kind: clickhouse
  config:
    # Endpoint — either `url` OR `host` (+ optional `http_port` / `tls`).
    url: http://localhost:8123
    # host: localhost
    # http_port: 8123
    # tls: false
    database: default
    user: default          # optional; sent as X-ClickHouse-User
    password: ${env:CH_PASSWORD}   # optional; sent as X-ClickHouse-Key
    query: SELECT id, email, updated_at FROM events
    batch_size: 1000       # records per StreamPage; 0 = whole result as one page

Do not append a FORMAT clause to query — the source sets the output format to JSONEachRow via the request settings.

Authentication

Username + password (ClickHouse native HTTP auth), sent as the X-ClickHouse-User / X-ClickHouse-Key headers (never URL query parameters, so credentials do not leak into request logs).

Incremental replication

Set replication to track a monotonically increasing column across runs. Only rows whose column value is strictly greater than the stored bookmark (or initial_value on the first run) are emitted, and the new maximum is persisted on the final page.

    replication:
      type: incremental
      column: updated_at
      initial_value: "1970-01-01 00:00:00"
    query: SELECT * FROM events WHERE updated_at > @bookmark

Put the literal @bookmark token in the WHERE clause to push the cursor down to the server (efficient); it is substituted as an injection-safe SQL literal. The source also filters client-side as a correctness backstop. If @bookmark is omitted the cursor is applied client-side only, so the server returns the whole result set on every run (correctness is preserved, but it is a full re-scan — a warning is logged).

You can inject parent-context values in a matrix child via {key} tokens; they are substituted as injection-safe SQL literals.

Dataset discovery / sharding

Not supported in v1.

License