Struct SparkSession 

pub struct SparkSession { /* private fields */ }

A Spark Connect session for interacting with a remote Spark cluster.

Mirrors pyspark.sql.SparkSession.

Implementations

impl SparkSession

pub fn add_tag(&self, tag: &str) -> Result<()>

Add a tag to be attached to all subsequent operations from this session.

Mirrors SparkSession.addTag. Tags cannot be empty or contain a comma.

pub fn remove_tag(&self, tag: &str)

Remove a previously added tag. Mirrors SparkSession.removeTag.

pub fn get_tags(&self) -> Vec<String>

Get the tags currently set on this session. Mirrors SparkSession.getTags.

pub fn clear_tags(&self)

Clear all tags set on this session. Mirrors SparkSession.clearTags.

pub fn register_progress_handler( &self, handler: impl Fn(&ExecutionProgress) + Send + Sync + 'static, ) -> u64

Register a handler invoked for every ExecutionProgress message received during query execution. Returns an id usable with SparkSession::remove_progress_handler. Mirrors registerProgressHandler.

pub fn remove_progress_handler(&self, id: u64)

Remove a progress handler by the id returned from SparkSession::register_progress_handler. Mirrors removeProgressHandler.

pub fn clear_progress_handlers(&self)

Remove all progress handlers. Mirrors clearProgressHandlers.

pub fn last_execution_info(&self) -> Option<ExecutionInfo>

The metrics captured from the most recent execution on this session, if any. Backs DataFrame::execution_info.

pub fn profile(&self) -> Vec<ObservedMetrics>

Raw observed metrics from the most recent execution only.

This is a low-level snapshot of the last execution’s observed metrics. For the profiler surface that mirrors SparkSession.profile (results accumulated across executions, with show/dump/clear), use SparkSession::profiler.

pub fn new_session(&self) -> SparkSession

Start a brand-new session over the same connection. Mirrors SparkSession.newSession — a fresh server-side session (new session id), with its own tags and plan-id counter.

pub fn clone_session(&self) -> SparkSession

Alias of SparkSession::new_session; mirrors the reference SparkSession.cloneSession, which creates a new session on the same client.

pub fn builder() -> SparkSessionBuilder

Create a builder for a new SparkSession.

pub fn range(&self, end: i64) -> Result<DataFrame>

Create a DataFrame representing a range of integers.

Mirrors pyspark.sql.SparkSession.range(start, end=None, step=1, numPartitions=None).

pub fn range_full( &self, start: i64, end: i64, step: i64, num_partitions: Option<i32>, ) -> Result<DataFrame>

Create a DataFrame representing a range with full parameters.

pub fn sql(&self, query: &str) -> Result<DataFrame>

Execute a SQL query and return a DataFrame.

Mirrors pyspark.sql.SparkSession.sql(sqlQuery).

pub fn sql_with_args( &self, query: &str, pos_args: Vec<Expression>, named_args: HashMap<String, Expression>, ) -> Result<DataFrame>

SQL with parameters: positional (pos_args) and/or named (named_args) expression bindings, mirroring SparkSession.sql(query, args=...).

pub fn create_dataframe( &self, rows: Vec<Row>, schema: DataType, ) -> Result<DataFrame>

Create a DataFrame from a collection of Rows and a schema.

Mirrors pyspark.sql.SparkSession.createDataFrame(rows, schema).

pub fn read(&self) -> DataFrameReader

Create a DataFrameReader for reading data from various sources.

Mirrors pyspark.sql.SparkSession.read.

pub fn read_stream(&self) -> DataStreamReader

Create a DataStreamReader for reading streaming data from various sources.

Mirrors pyspark.sql.SparkSession.readStream.

pub fn streams(&self) -> StreamingQueryManager

Get the streaming query manager.

Mirrors pyspark.sql.SparkSession.streams.

pub fn catalog(&self) -> Catalog

Get the catalog for this session.

Mirrors pyspark.sql.SparkSession.catalog.

pub fn register_java_function( &self, name: &str, java_class_name: &str, return_type_ddl: Option<&str>, aggregate: bool, ) -> Result<()>

Register a Java UDF/UDAF by class name, mirroring the reference client.register_java(name, javaClassName, return_type, aggregate) used by UDFRegistration.registerJavaFunction / registerJavaUDAF: builds a CommonInlineUserDefinedFunction carrying a JavaUDF and sends it as the RegisterFunction command. return_type_ddl is only used for non-aggregate functions (matching the reference, which omits the output type when aggregate).

pub fn conf(&self) -> RuntimeConf

Get the runtime configuration for this session.

Mirrors pyspark.sql.SparkSession.conf.

pub fn tvf(&self) -> TableValuedFunction

Get table-valued functions for this session.

Mirrors pyspark.sql.SparkSession.tvf.

pub fn session_id(&self) -> &str

The session ID of this session.

Mirrors pyspark.sql.SparkSession.session_id / the connect client session id.

pub fn version(&self) -> Result<String>

Return the Spark version of the connected server.

Mirrors pyspark.sql.SparkSession.version.

pub fn table(&self, table_name: &str) -> Result<DataFrame>

Return the DataFrame for the given table/view.

Mirrors pyspark.sql.SparkSession.table(tableName).

pub fn empty_data_frame(&self) -> Result<DataFrame>

Return an empty DataFrame with no rows and an empty schema.

Mirrors pyspark.sql.SparkSession.createDataFrame([], StructType([])) / SparkSession.emptyDataFrame.

pub fn interrupt_all(&self) -> Result<Vec<String>>

Interrupt all operations of this session.

Mirrors pyspark.sql.SparkSession.interruptAll(). Returns interrupted operation ids.

pub fn interrupt_tag(&self, tag: &str) -> Result<Vec<String>>

Interrupt all operations of this session with the given tag.

Mirrors pyspark.sql.SparkSession.interruptTag(tag).

pub fn interrupt_operation(&self, operation_id: &str) -> Result<Vec<String>>

Interrupt the operation with the given operation id.

Mirrors pyspark.sql.SparkSession.interruptOperation(opId).

pub fn add_artifacts(&self, paths: &[&str]) -> Result<()>

Add local files as artifacts to the session (e.g. .py, .jar, .zip).

Mirrors pyspark.sql.SparkSession.addArtifacts(*path).

pub fn add_artifact(&self, path: &str) -> Result<()>

Add a single local file as an artifact to the session.

Mirrors pyspark.sql.SparkSession.addArtifact(path).

pub fn copy_from_local_to_fs( &self, local_path: &str, dest_path: &str, ) -> Result<()>

Copy a local file to the driver’s filesystem at dest_path.

Mirrors pyspark.sql.SparkSession.copyFromLocalToFs: uploads the file as a forward_to_fs/<dest_path> artifact, which the server writes to dest_path.

pub fn register_function( &self, udf: CommonInlineUserDefinedFunctionExpression, ) -> Result<()>

Register a user-defined function on the session so it can be referenced by name in SQL / expressions.

Mirrors the server-side effect of pyspark.sql.SparkSession.udf.register / udtf.register: the UDF is cloudpickled on the client (see crate::udf) and sent as a RegisterFunction command.

pub fn build_resource_profile(&self, profile: &ResourceProfile) -> Result<i32>

Build and register a ResourceProfile with the server.

Sends a CreateResourceProfileCommand to the server with the specified executor and task resource requests, and returns the server-assigned profile id. The profile can then be used with DataFrame.withResources(profile_id).

Mirrors pyspark.sql.SparkSession._build_resource_profile (internal).

pub fn register_data_source( &self, data_source: CommonInlineUserDefinedDataSourceExpression, ) -> Result<()>

Register a user-defined data source on the session so it can be referenced in SQL queries.

Mirrors the server-side effect of pyspark.sql.SparkSession.dataSource.register. The data source is cloudpickled on the Python client and sent as a RegisterDataSource command. Since Rust cannot cloudpickle Python classes, the command bytes must be prepared on the client (typically by a Python wrapper).

pub fn profiler(&self) -> Arc<ProfilerCollector>

Get the profiler collector for this session.

Mirrors the client-visible surface of SparkSession.profile. Profile data is accumulated across query executions and can be shown, dumped, or cleared via the returned collector. Profile data is populated by the server only when UDF profiling is enabled via spark.python.profile* or spark.sql.pyspark.udf.profiler configuration.

pub fn stop(&self) -> Result<()>

Stop this Spark session.

pub fn is_stopped(&self) -> bool

Whether stop() has been called on this session. Mirrors pyspark.sql.connect.session.SparkSession.is_stopped.

Trait Implementations

impl Clone for SparkSession

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.