faucet-source-databricks

Databricks SQL query source for the faucet-stream ecosystem. Runs a SQL statement against a Databricks SQL Warehouse via the Statement Execution API (plain REST — no JDBC/ODBC driver, no Python) and streams the result rows as typed JSON objects.

This is the query-results read path for Databricks — joins, aggregates, filtered extracts. For full-table lakehouse scans and the write path, use the Delta Lake connectors (faucet-source-delta / faucet-sink-delta); a warehouse INSERT/MERGE sink is intentionally not provided (slow, INSERT-bound, and forces billed compute).

Highlights

• Async statement lifecycle — submit → poll until terminal → stream result chunks (INLINE + JSON_ARRAY), following next_chunk_internal_link.
• Type-aware decode from the response manifest column schema (every JSON_ARRAY cell is a string; decoded per type_name to typed JSON, with DECIMAL/large LONG preserved losslessly as strings).
• Incremental replication — a bookmark column + a ${bookmark} token bound as a server-side named parameter, plus a client-side filter backstop.
• Bearer auth (PAT / OAuth M2M), inline or via the shared auth: catalog.
• Arrow-native fetch — behind the arrow feature, arrow_native: true fetches results as EXTERNAL_LINKS + ARROW_STREAM and decodes each chunk as an Arrow IPC stream, enabling the columnar fast path and skipping the per-cell JSON decode on the row path. See Arrow columnar (Parquet) mode.

Configuration

pipeline:
  source:
    type: databricks
    config:
      workspace_url: https://dbc-xxxx.cloud.databricks.com
      warehouse_id: 0123456789abcdef
      sql: SELECT id, ts FROM events WHERE ts > ${bookmark}
      auth: { type: pat, config: { token: "${env:DATABRICKS_TOKEN}" } }
      replication: { type: incremental, column: ts, initial_value: "2026-01-01" }

Arrow columnar (Parquet) mode

Behind the crate-local arrow Cargo feature, setting arrow_native: true submits the statement with EXTERNAL_LINKS disposition and ARROW_STREAM format; each result chunk's presigned link is fetched and decoded as an Arrow IPC stream rather than a JSON_ARRAY. This has two effects:

• the opt-in columnar fast path (RFC 0002 / #375) — when the sink is also Arrow-native (the Parquet or Delta Lake sink) and no Value-shaped transform is configured, records move end-to-end as Arrow RecordBatches with no serde_json::Value materialization; and
• a faster row path — even into a JSON sink, decoding Arrow batches skips the per-cell string→JSON decode the default INLINE + JSON_ARRAY path pays.

arrow_native: true requires the arrow feature and only works with replication: full — config validation rejects arrow_native combined with incremental replication, because the columnar path does not run the per-row client-side incremental filter. The default (arrow_native: false) keeps the unchanged INLINE + JSON_ARRAY row path.

# databricks(arrow) → parquet — runs Arrow end-to-end
pipeline:
  source:
    type: databricks
    config:
      workspace_url: https://dbc-xxxx.cloud.databricks.com
      warehouse_id: 0123456789abcdef
      sql: SELECT id, ts, payload FROM events
      auth: { type: pat, config: { token: "${env:DATABRICKS_TOKEN}" } }
      arrow_native: true   # requires the `arrow` feature; replication must be `full`
  sink:
    type: parquet
    config:
      path: ./out/

Enable it with cargo add faucet-source-databricks --features arrow (library) or cargo install faucet-cli --features "source-databricks,arrow" (CLI).

Out of scope (v1)

discover() via INFORMATION_SCHEMA, Unity Catalog Volumes, and a Databricks SQL sink (use the Delta sink). (The EXTERNAL_LINKS large-result disposition is available via arrow_native — see above.)

License: MIT OR Apache-2.0.