spark-connect

An idiomatic, SQL-first Rust client for Apache Spark Connect.

This crate provides a fully asynchronous, strongly typed API for interacting with a remote Spark Connect server over gRPC.

It allows you to build and execute SQL queries, bind parameters safely, and collect Arrow RecordBatch results - just like any other SQL toolkit - all in native Rust.

✨ Features

⚙️ Spark-compatible connection builder (sc://host:port format);
🪶 Async execution using tokio and tonic;
🧩 Parameterized queries;
🧾 Arrow-native results returned as Vec<RecordBatch>;

Getting Started

use spark_connect::SparkSessionBuilder;

# #[tokio::main]
# async fn main() -> Result<(), Box<dyn std::error::Error>> {
// 1️⃣ Connect to a Spark Connect endpoint
let session = SparkSessionBuilder::new("sc://localhost:15002")
    .build()
    .await?;

// 2️⃣ Execute a simple SQL query and receive a Vec<RecordBatches>
let batches = session
    .query("SELECT ? AS rule, ? AS text")
    .bind(42)
    .bind("world")
    .execute()
    .await?;

# Ok(())
# }

It's that simple!

🧩 Parameterized Queries

Behind the scenes, the [SparkSession::query] method uses the [ToLiteral] trait to safely bind parameters before execution:

use spark_connect::ToLiteral;
 
// This is
 
let batches = session
    .query("SELECT ? AS id, ? AS text")
    .bind(42)
    .bind("world")
    .await?;

// the same as this

let lazy_plan = session.sql(
    "SELECT ? AS id, ? AS text",
    vec![42.to_literal(), "world".to_literal()]
).await?;
let batches = session.collect(lazy_plan);

😴 Lazy Execution

The biggest advantage to using the sql() method instead of query() is lazy execution - queries can be lazily evaluated and collected afterwards. If you're coming from PySpark or Scala, this should be the familiar interface.

🧠 Concepts

SparkSession — the main entry point for executing SQL queries and managing a session.
SparkClient — low-level gRPC client (used internally).
SqlQueryBuilder — helper for binding parameters and executing queries.

⚙️ Requirements

A running Spark Connect server (Spark 3.4+);
Network access to the configured sc:// endpoint;
tokio runtime.

🔒 Example Connection Strings

sc://localhost:15002
sc://spark-cluster:15002/?user_id=francisco
sc://10.0.0.5:15002;session_id=abc123;user_agent=my-app

📘 Learn More

🙏 Acknowledgements

This project takes heavy inspiration from the spark-connect-rs project, and would've been much harder without it!

This project is not affiliated with, endorsed by, or sponsored by the Apache Software Foundation. “Apache”, “Apache Spark”, and “Spark Connect” are trademarks of the Apache Software Foundation.

spark-connect 0.1.1