Crate spark_connect 

Pure-Rust Spark Connect DataFrame client mirroring the PySpark API surface.

This crate provides the core data types and operations for interacting with Apache Spark via the Spark Connect protocol.

Re-exports

pub use column::col;

pub use column::lit;

pub use column::lit_boolean;

pub use column::lit_double;

pub use column::lit_string;

pub use column::Column;

pub use dataframe::DataFrame;

pub use dataframe::LocalRowIterator;

pub use datasource::CommonInlineUserDefinedDataSourceExpression;

pub use datasource::PythonDataSourcePayload;

pub use expression::Alias;

pub use expression::CaseWhen;

pub use expression::Cast;

pub use expression::ColumnReference;

pub use expression::Expression;

pub use expression::LiteralExpression;

pub use expression::SortOrder;

pub use expression::UnresolvedFunction;

pub use group::CoGroupedData;

pub use group::GroupedData;

pub use merge::MergeIntoWriter;

pub use observation::Observation;

pub use profiler::ProfilerCollector;

pub use readwriter::DataFrameReader;

pub use readwriter::DataFrameWriter;

pub use readwriter::DataFrameWriterV2;

pub use readwriter::ReadType;

pub use readwriter::SaveMode;

pub use readwriter::TableSaveMethod;

pub use resource::ExecutorResourceRequests;

pub use resource::ResourceProfile;

pub use resource::ResourceProfileBuilder;

pub use resource::TaskResourceRequests;

pub use session::SparkSession;

pub use session::SparkSessionBuilder;

pub use storage::StorageLevelExt;

pub use streaming::DataStreamReader;

pub use streaming::DataStreamWriter;

pub use streaming::StreamingQuery;

pub use streaming::StreamingQueryManager;

pub use streaming::Trigger;

pub use types::DataType;

pub use types::StructField;

pub use window::FrameBound;

pub use window::FrameType;

pub use window::Window;

pub use window::WindowSpec;

Modules

catalog

Catalog API mirroring pyspark.sql.connect.catalog.Catalog.

column

Column API mirroring PySpark’s pyspark.sql.connect.column.Column.

conf

RuntimeConf implementation mirroring pyspark.sql.conf.RuntimeConfig.

dataframe

DataFrame implementation mirroring pyspark.sql.DataFrame.

datasource

Data source registration support.

expression

Expression tree mirroring PySpark’s pyspark.sql.connect.expressions.

functions

SQL functions mirroring pyspark.sql.connect.functions.builtin.

group

GroupedData implementation for aggregations.

merge

MergeIntoWriter mirroring pyspark.sql.connect.merge.MergeIntoWriter.

ml

ML module mirroring pyspark.ml.connect.base and related ML classes.

observation

Observation implementation for collecting metrics.

pipelines

Spark Declarative Pipelines (SDP) command execution.

plan

Logical plan nodes mirroring pyspark.sql.connect.plan.

profiler

Client-side profiler collector for UDF and plan profiling results.

readwriter

DataFrameReader and DataFrameWriter implementation mirroring pyspark.sql.connect.readwriter.

resource

Resource profile for specifying executor and task resource requirements.

row

Row type mirroring pyspark.sql.Row.

session

SparkSession implementation mirroring pyspark.sql.SparkSession.

storage

Storage-level presets, mirroring pyspark.StorageLevel.

streaming

Structured Streaming support mirroring pyspark.sql.connect.streaming.

table_arg

Table argument support mirroring pyspark.sql.connect.table_arg.TableArg.

tvf

Table-valued functions (TVF) implementation mirroring pyspark.sql.connect.tvf.TableValuedFunction.

types

DataType type system mirroring PySpark’s pyspark.sql.types.

udf

User-defined function (UDF) support.

window

Window functions and specifications mirroring PySpark’s pyspark.sql.window.

Structs

SparkError

Re-exported so fallible APIs (e.g. the #[spark_wasm_udf]-generated UDF constructors) can name the client’s Result/error types. A structured error carrying a PySpark error class and its message parameters.

StorageLevel

Re-exported so persist(...) / storage_level() have a public storage-level type. StorageLevel for persisting Datasets/Tables.

Type Aliases

Result

Re-exported so fallible APIs (e.g. the #[spark_wasm_udf]-generated UDF constructors) can name the client’s Result/error types.