spark-connect

An idiomatic, SQL-first Rust client for Apache Spark Connect.

This crate is an async layer over the official apache-spark-connect client, whose API is synchronous. It brings the whole official API into async code and adds a sqlx-style interface for binding parameters safely.

Changelog

✨ Features

• ⚙️ Spark-compatible connection builder (sc://host:port format);
• 🪶 Async execution on tokio, for the whole official API;
• 🧩 Parameterized queries, bound through the ToLiteral trait;
• 🧾 Results as Arrow RecordBatches, typed Rows, or a row stream;

Getting Started

use spark_connect::SparkSession;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1️⃣ Connect to a Spark Connect endpoint
    let session = SparkSession::builder()
        .remote("sc://localhost:15002")
        .get_or_create()
        .await?;

    // 2️⃣ Execute a simple SQL query and receive a Vec<RecordBatch>
    let batches = session
        .query("SELECT ? AS rule, ? AS text")
        .bind(42)
        .bind("world")
        .execute()
        .await?;

    Ok(())
}

It's that simple!

🧩 Parameterized Queries

The SparkSession::query method uses the ToLiteral trait to bind each parameter as a Spark literal, so values never reach the server as SQL text:

// This
let batches = session
    .query("SELECT ? AS id, ? AS text")
    .bind(42)
    .bind("world")
    .execute()
    .await?;

// is the same as this
use spark_connect::{prelude::*, Expression};

let batches = session
    .run(|spark| {
        spark
            .sql_with_args(
                "SELECT ? AS id, ? AS text",
                vec![
                    Expression::Literal(42.to_literal()),
                    Expression::Literal("world".to_literal()),
                ],
                Default::default(),
            )?
            .collect_record_batches()
    })
    .await?;

bind accepts Rust primitives, String/&str, Vec<u8> and, with the chrono feature, NaiveDate and NaiveDateTime. Pass a LiteralExpression directly for decimals, arrays, maps, structs and typed nulls.

Named parameters and the rest of the official SQL API are reached through sql_with_args, as above.

🧰 DataFrames

The official client's DataFrame API is used as it is: transformations only build a plan, and each action has an _async counterpart in the prelude, run off the async executor:

use spark_connect::prelude::*;

let df = spark.sql("SELECT * FROM people")?.filter(col("age").gt(lit(17)));

let adults = df.count_async().await?;
let rows = df.collect_async().await?;

This crate re-exports the official crate's items - DataFrame, Column, functions, col, lit, ... - alongside its own. Anything else in the official API is reachable through SparkSession::run.

⚖️ Trade-offs

The official client blocks on a runtime of its own, so every call that reaches the server runs on tokio's blocking thread pool. In practice:

• Dropping a future does not cancel the query. A timed-out or abandoned action runs to completion on the server and its result is discarded. Use SparkSession::interrupt_all or SparkSession::interrupt_tag to stop work on the server.
• Each in-flight action holds a blocking thread. Tokio's pool allows 512 by default, far above typical Spark concurrency.
• The official synchronous actions remain callable - df.collect() next to df.collect_async() - and panic when called from an async task.
• Large results can be streamed row by row with to_local_iterator_async.

🧠 Concepts

• SparkSession - the main entry point for executing SQL queries and managing a session.
• SqlQueryBuilder - helper for binding parameters and executing queries.
• ext - async counterparts of the official types' actions.

⚙️ Requirements

• A running Spark Connect server (Spark 4.0+);
• Network access to the configured sc:// endpoint;
• tokio runtime;
• protoc at build time.

Installing protoc

apache-spark-connect-proto compiles the Connect .proto files while it builds, using the Protocol Buffers compiler:

apt-get install -y protobuf-compiler  # Debian, Ubuntu
brew install protobuf                 # macOS
winget install protobuf               # Windows

If it is not on the PATH, point PROTOC at it:

PROTOC=/path/to/protoc cargo build

In GitHub Actions:

- uses: arduino/setup-protoc@v3
  with:
    repo-token: ${{ secrets.GITHUB_TOKEN }}

🔒 Example Connection Strings

sc://localhost:15002
sc://spark-cluster:15002/;user_id=francisco
sc://10.0.0.5:15002/;session_id=abc123;user_agent=my-app

📘 Learn More

• apache-spark-connect API reference - the DataFrame, column and function APIs this crate builds on;
• Apache Spark Connect documentation;
• Spark Connect client connection string.

© 2025 Francisco A. B. Sampaio. Licensed under the Apache License 2.0.

This project is not affiliated with, endorsed by, or sponsored by the Apache Software Foundation. “Apache”, “Apache Spark”, and “Spark Connect” are trademarks of the Apache Software Foundation.