faucet-source-databricks

Databricks SQL query source for the faucet-stream ecosystem. Runs a SQL statement against a Databricks SQL Warehouse via the Statement Execution API (plain REST — no JDBC/ODBC driver, no Python) and streams the result rows as typed JSON objects.

This is the query-results read path for Databricks — joins, aggregates, filtered extracts. For full-table lakehouse scans and the write path, use the Delta Lake connectors (faucet-source-delta / faucet-sink-delta). To load into a Databricks SQL warehouse (Unity Catalog permissions, COPY INTO, MERGE, exactly-once), use faucet-sink-databricks.

Highlights

• Async statement lifecycle — submit → poll until terminal → stream result chunks (EXTERNAL_LINKS + JSON_ARRAY: each chunk is downloaded from its presigned URL without the workspace token), following the chunk links, so results of any size stream — INLINE results are capped at 25 MiB by Databricks.
• Type-aware decode from the response manifest column schema (every JSON_ARRAY cell is a string; decoded per type_name to typed JSON, with DECIMAL/large LONG preserved losslessly as strings).
• Incremental replication — a bookmark column + a ${bookmark} token bound as a server-side named parameter, plus a client-side filter backstop.
• Bearer auth (PAT / OAuth M2M), inline or via the shared auth: catalog. The auth and Statement Execution API client live in faucet-common-databricks (shared with the sink). This crate keeps its own DatabricksAuth (same wire shape, converted with From) so its public API is unchanged.
• Warm-up tolerant — a submit refused with 429/503 (a serverless warehouse starting) is retried with backoff honouring Retry-After, polls retry transient 5xx, and a 401 makes a shared provider mint a fresh token once.
• Arrow-native fetch — behind the arrow feature, arrow_native: true fetches results as EXTERNAL_LINKS + ARROW_STREAM and decodes each chunk as an Arrow IPC stream, enabling the columnar fast path and skipping the per-cell JSON decode on the row path. See Arrow columnar (Parquet) mode.

Configuration

pipeline:
  source:
    type: databricks
    config:
      workspace_url: https://dbc-xxxx.cloud.databricks.com
      warehouse_id: 0123456789abcdef
      sql: SELECT id, ts FROM events WHERE ts > ${bookmark}
      auth: { type: pat, config: { token: "${env:DATABRICKS_TOKEN}" } }
      replication: { type: incremental, column: ts, initial_value: "2026-01-01" }

Arrow columnar (Parquet) mode

Behind the crate-local arrow Cargo feature, setting arrow_native: true submits the statement with EXTERNAL_LINKS disposition and ARROW_STREAM format; each result chunk's presigned link is fetched and decoded as an Arrow IPC stream rather than a JSON_ARRAY. This has two effects:

• the opt-in columnar fast path (RFC 0002 / #375) — when the sink is also Arrow-native (the Parquet or Delta Lake sink) and no Value-shaped transform is configured, records move end-to-end as Arrow RecordBatches with no serde_json::Value materialization; and
• a faster row path — even into a JSON sink, decoding Arrow batches skips the per-cell string→JSON decode the default JSON_ARRAY path pays.

arrow_native: true requires the arrow feature and only works with replication: full — config validation rejects arrow_native combined with incremental replication, because the columnar path does not run the per-row client-side incremental filter. The default (arrow_native: false) keeps the JSON_ARRAY row path.

# databricks(arrow) → parquet — runs Arrow end-to-end
pipeline:
  source:
    type: databricks
    config:
      workspace_url: https://dbc-xxxx.cloud.databricks.com
      warehouse_id: 0123456789abcdef
      sql: SELECT id, ts, payload FROM events
      auth: { type: pat, config: { token: "${env:DATABRICKS_TOKEN}" } }
      arrow_native: true   # requires the `arrow` feature; replication must be `full`
  sink:
    type: file
    config:
      path: ./out/
      format: parquet

Enable it with cargo add faucet-source-databricks --features arrow (library) or cargo install faucet-cli --features "source-databricks,arrow" (CLI).

Out of scope (v1)

discover() via INFORMATION_SCHEMA, Unity Catalog Volumes, and a Databricks SQL sink (use the Delta sink).

License: MIT OR Apache-2.0.