Struct DataFrame 

pub struct DataFrame { /* private fields */ }

A Spark DataFrame, lazily evaluated.

Mirrors pyspark.sql.DataFrame.

Implementations

impl DataFrame

pub fn select<C: Into<Column>>( &self, columns: impl IntoIterator<Item = C>, ) -> DataFrame

Select specific columns.

pub fn filter(&self, condition: Column) -> DataFrame

Filter rows by a condition.

pub fn where_(&self, condition: Column) -> DataFrame

Alias for filter().

pub fn with_column(&self, name: &str, col: Column) -> DataFrame

Add or replace a column.

pub fn with_columns(&self, columns: Vec<(String, Column)>) -> DataFrame

Add or replace multiple columns.

pub fn with_column_renamed(&self, existing: &str, new: &str) -> DataFrame

Rename a column.

pub fn with_columns_renamed(&self, renames: Vec<(String, String)>) -> DataFrame

Rename multiple columns.

pub fn drop(&self, columns: Vec<&str>) -> DataFrame

Drop columns.

pub fn limit(&self, n: i32) -> DataFrame

Limit the number of rows.

pub fn offset(&self, n: i32) -> DataFrame

Skip the first n rows.

pub fn tail(&self, n: i32) -> DataFrame

Get the last n rows.

pub fn distinct(&self) -> DataFrame

Remove duplicate rows.

pub fn drop_duplicates(&self, column_names: Option<Vec<&str>>) -> DataFrame

Remove duplicate rows, optionally on specific columns.

pub fn sort(&self, columns: Vec<Expression>) -> DataFrame

Sort rows.

pub fn order_by(&self, columns: Vec<Expression>) -> DataFrame

Alias for sort().

pub fn join( &self, right: &DataFrame, on: Option<Column>, join_type: JoinType, ) -> DataFrame

Join with another DataFrame.

pub fn join_using<S: Into<String>>( &self, right: &DataFrame, using_columns: impl IntoIterator<Item = S>, join_type: JoinType, ) -> DataFrame

Join with another DataFrame using column names (a name-based/“using” join).

pub fn nearest_by_join( &self, other: &DataFrame, ranking_expression: Column, num_results: i32, mode: &str, direction: &str, join_type: &str, ) -> DataFrame

Nearest-by join: for each left row, the num_results nearest right rows ranked by ranking_expression. Mirrors DataFrame.nearestByJoin.

mode ∈ {“approx”,“exact”}, direction ∈ {“distance”,“similarity”}, join_type ∈ {“inner”,“leftouter”}.

pub fn cross_join(&self, right: &DataFrame) -> DataFrame

Cross join.

pub fn lateral_join( &self, right: &DataFrame, on: Option<Column>, join_type: JoinType, ) -> DataFrame

Lateral join with another DataFrame (a LATERAL correlated subquery join).

Mirrors pyspark.sql.DataFrame.lateralJoin.

pub fn union(&self, other: &DataFrame) -> DataFrame

Union with another DataFrame.

pub fn union_by_name(&self, other: &DataFrame) -> DataFrame

Union by name.

pub fn union_by_name_opt( &self, other: &DataFrame, allow_missing_columns: bool, ) -> DataFrame

unionByName with the allowMissingColumns option (columns present in only one side are filled with null rather than rejected). Mirrors DataFrame.unionByName(other, allowMissingColumns=False).

pub fn intersect(&self, other: &DataFrame) -> DataFrame

Intersect with another DataFrame.

pub fn subtract(&self, other: &DataFrame) -> DataFrame

Subtract (except) another DataFrame.

pub fn repartition(&self, num_partitions: i32) -> DataFrame

Repartition.

pub fn coalesce(&self, num_partitions: i32) -> DataFrame

Coalesce.

pub fn hint<S: Into<String>>( &self, name: &str, parameters: impl IntoIterator<Item = S>, ) -> DataFrame

Add a hint.

pub fn broadcast(&self) -> DataFrame

Marks a DataFrame as eligible for broadcast join (smaller table). Mirrors pyspark.sql.functions.broadcast.

pub fn to_df(&self, column_names: Vec<&str>) -> DataFrame

Convert to DataFrame with new column names.

pub fn alias(&self, alias: &str) -> DataFrame

Alias this DataFrame.

pub fn map_in_pandas( &self, func: CommonInlineUserDefinedFunctionExpression, is_barrier: bool, ) -> DataFrame

Map over each partition with a pandas UDF (DataFrame.mapInPandas).

The func (built by the Python side, cloudpickled with eval type SQL_MAP_PANDAS_ITER_UDF) is applied to iterators of pandas DataFrames.

pub fn map_in_arrow( &self, func: CommonInlineUserDefinedFunctionExpression, is_barrier: bool, ) -> DataFrame

Map over each partition with an Arrow UDF (DataFrame.mapInArrow).

pub fn foreach( &self, func: CommonInlineUserDefinedFunctionExpression, ) -> Result<()>

Apply a function to each row for its side effects (DataFrame.foreach).

Backed by an Arrow map partition; results are forced and discarded.

pub fn foreach_partition( &self, func: CommonInlineUserDefinedFunctionExpression, ) -> Result<()>

Apply a function to each partition for its side effects (DataFrame.foreachPartition).

pub fn sample(&self, fraction: f64, seed: Option<i64>) -> DataFrame

Sample rows.

pub fn sample_opt( &self, fraction: f64, with_replacement: bool, seed: Option<i64>, ) -> DataFrame

sample with the withReplacement option. Mirrors DataFrame.sample(withReplacement, fraction, seed).

pub fn group_by<C: Into<Column>>( &self, group_cols: impl IntoIterator<Item = C>, ) -> GroupedData

Group by columns for aggregation.

pub fn collect(&self) -> Result<Vec<Row>>

Collect all rows into memory.

pub fn to_local_iterator( &self, prefetch_partitions: bool, ) -> Result<LocalRowIterator>

Return an iterator that lazily streams rows from the server.

Mirrors pyspark.sql.DataFrame.toLocalIterator(prefetchPartitions=False). Unlike collect(), which buffers all results in memory, this returns an iterator that yields Row objects as the server streams them, consuming minimal memory.

Arguments

• prefetch_partitions - If true, a background task fetches the next batch from the server while the caller consumes the current one (one batch buffered ahead), overlapping network I/O with row processing. If false, each batch is fetched on demand only once the previous batch is exhausted.

pub fn execution_info(&self) -> Result<ExecutionInfo>

Execution metrics collected during the most recent action on this DataFrame’s session. Mirrors pyspark.sql.DataFrame.executionInfo.

The metrics reflect the session’s most recent execution; call this right after an action (e.g. collect/count/show).

pub fn collect_record_batches(&self) -> Result<Vec<RecordBatch>>

Collect all data as Arrow RecordBatches.

Streams execution results from the server and decodes Arrow IPC batches, returning the raw RecordBatches without converting to Rows. This is the foundation for to_datafusion() and to_polars() conversions.

pub fn count(&self) -> Result<i64>

Get the count of rows.

Mirrors pyspark.sql.DataFrame.count() = groupBy().count().collect()[0][0]: a global count aggregate is pushed to the server, which returns a single row, rather than streaming every row back to the client just to count them.

pub fn show(&self, n: usize) -> Result<()>

Show the first n rows.

pub fn schema(&self) -> Result<DataType>

Get the schema of this DataFrame.

pub fn first(&self) -> Result<Option<Row>>

Get the first row.

pub fn head(&self) -> Result<Option<Row>>

Alias for first().

pub fn take(&self, n: usize) -> Result<Vec<Row>>

Get the first n rows.

pub fn is_empty(&self) -> Result<bool>

Check if the DataFrame is empty.

pub fn columns(&self) -> Result<Vec<String>>

Get column names.

pub fn write(&self) -> DataFrameWriter

Create a DataFrameWriter for writing this DataFrame to various destinations.

Mirrors pyspark.sql.DataFrame.write.

pub fn write_to(&self, table_name: &str) -> DataFrameWriterV2

Create a DataFrameWriterV2 for the v2 write API.

Mirrors pyspark.sql.DataFrame.writeTo.

pub fn merge_into(&self, table: &str, condition: Column) -> MergeIntoWriter

Merge a set of updates, insertions, and deletions into a target table.

Mirrors pyspark.sql.DataFrame.mergeInto: returns a crate::merge::MergeIntoWriter on which when_matched / when_not_matched / when_not_matched_by_source clauses are added before calling merge().

pub fn write_stream(&self) -> DataStreamWriter

Create a DataStreamWriter for writing this streaming DataFrame to various sinks.

Mirrors pyspark.sql.DataFrame.writeStream.

pub fn cache(&self) -> Result<DataFrame>

Cache this DataFrame with the default MEMORY_AND_DISK_DESER storage level.

Mirrors pyspark.sql.DataFrame.cache().

pub fn persist(&self, storage_level: StorageLevel) -> Result<DataFrame>

Persist this DataFrame with the given storage level.

Mirrors pyspark.sql.DataFrame.persist(storageLevel).

pub fn unpersist(&self, blocking: bool) -> Result<DataFrame>

Remove this DataFrame from cache. Mirrors DataFrame.unpersist(blocking).

pub fn checkpoint(&self) -> Result<DataFrame>

Checkpoint this DataFrame to disk.

pub fn local_checkpoint(&self) -> Result<DataFrame>

Create a local checkpoint of this DataFrame.

pub fn create_temp_view(&self, name: &str) -> Result<()>

Create a temporary view for this DataFrame.

pub fn create_or_replace_temp_view(&self, name: &str) -> Result<()>

Create or replace a temporary view for this DataFrame.

pub fn create_global_temp_view(&self, name: &str) -> Result<()>

Create a global temporary view for this DataFrame.

pub fn create_or_replace_global_temp_view(&self, name: &str) -> Result<()>

Create or replace a global temporary view for this DataFrame.

pub fn explain(&self) -> Result<()>

Print the execution plan to the console. Mirrors pyspark.sql.DataFrame.explain (was previously a no-op that ran the query relation instead of an AnalyzePlan).

pub fn explain_mode(&self, mode: &str) -> Result<()>

Print the execution plan in a specific mode. Mirrors the mode argument of pyspark.sql.DataFrame.explain: one of “simple”, “extended”, “codegen”, “cost”, “formatted” (case-insensitive).

pub fn with_watermark( &self, time_column: &str, delay_threshold: &str, ) -> DataFrame

Add a watermark to this DataFrame for event-time based windows.

pub fn repartition_by_range( &self, num_partitions: i32, columns: Vec<Expression>, ) -> DataFrame

Repartition this DataFrame by range.

pub fn repartition_by_expressions( &self, num_partitions: i32, columns: Vec<Expression>, ) -> DataFrame

Repartition into num_partitions by hashing the given column expressions. Mirrors df.repartition(numPartitions, *cols).

pub fn to_schema(&self, column_names: Vec<&str>) -> DataFrame

Alias for to_df().

pub fn melt( &self, id_vars: Vec<&str>, value_vars: Option<Vec<&str>>, var_name: &str, value_name: &str, ) -> DataFrame

Melt (unpivot) this DataFrame.

pub fn input_files(&self) -> Result<Vec<String>>

Get the input files for this DataFrame. Mirrors pyspark.sql.DataFrame.inputFiles.

pub fn observe(&self, name: &str, exprs: Vec<Expression>) -> DataFrame

Observe metrics on this DataFrame.

pub fn stat(&self) -> StatFunctions

Get stat functions.

pub fn na(&self) -> NaFunctions

Returns a crate::group::NaFunctions for handling missing values.

Mirrors pyspark.sql.DataFrame.na.

pub fn agg(&self, expressions: Vec<Expression>) -> DataFrame

Perform aggregation without grouping.

pub fn select_expr(&self, exprs: Vec<&str>) -> DataFrame

Select with SQL expressions, mirroring DataFrame.selectExpr.

Each string is parsed as a SQL expression (e.g. "id + 1 AS x"), not treated as a bare column name - so it must go through functions::expr (an ExpressionString the server parses), not col (an unresolved attribute, which made selectExpr("id + 1 AS x") fail to resolve).

pub fn fillna(&self, value: i64, subset: Option<Vec<&str>>) -> DataFrame

Fill NA values with an integer.

pub fn fillna_double(&self, value: f64, subset: Option<Vec<&str>>) -> DataFrame

Fill NA values with a double (e.g. a fractional fill into a double column).

pub fn fillna_string(&self, value: &str, subset: Option<Vec<&str>>) -> DataFrame

Fill NA values with a string.

pub fn fillna_bool(&self, value: bool, subset: Option<Vec<&str>>) -> DataFrame

Fill NA values with a boolean.

pub fn fillna_value(&self, value: Value, subset: Option<Vec<&str>>) -> DataFrame

Fill NA values with a typed crate::row::Value (Long/Double/String/Bool/…).

pub fn fillna_map(&self, pairs: Vec<(String, Value)>) -> DataFrame

Fill NA values per column from (column, value) pairs. Mirrors df.fillna({col: value, ...}).

pub fn dropna( &self, how: Option<&str>, thresh: Option<i32>, subset: Option<Vec<&str>>, ) -> DataFrame

Drop NA values.

pub fn replace( &self, to_replace: Vec<(String, String)>, subset: Option<Vec<&str>>, ) -> DataFrame

Replace values.

pub fn describe(&self, columns: Vec<&str>) -> DataFrame

Describe this DataFrame (show statistics).

pub fn summary(&self, percentiles: Vec<&str>) -> DataFrame

Get summary statistics.

pub fn col_regex(&self, col_name: &str) -> DataFrame

Select columns by regex pattern.

pub fn metadata_column(&self, name: &str) -> Column

Select a metadata column by name. Mirrors pyspark.sql.DataFrame.metadataColumn.

pub fn rollup<C: Into<Column>>( &self, group_cols: impl IntoIterator<Item = C>, ) -> GroupedData

Group by with rollup.

pub fn cube<C: Into<Column>>( &self, group_cols: impl IntoIterator<Item = C>, ) -> GroupedData

Group by with cube.

pub fn grouping_sets(&self, group_cols: Vec<Vec<Column>>) -> GroupedData

Group by with grouping sets. Each inner Vec<Column> is one grouping set; the sets are preserved on the wire (GROUP_TYPE_GROUPING_SETS + the grouping_sets field) rather than flattened into a single group-by.

pub fn sort_within_partitions(&self, columns: Vec<Expression>) -> DataFrame

Sort within partitions (local sort).

pub fn drop_duplicates_within_watermark( &self, column_names: Option<Vec<&str>>, ) -> DataFrame

Drop duplicates within a watermark.

pub fn transform<F>(&self, f: F) -> DataFrame

where F: Fn(&DataFrame) -> DataFrame,

Apply a transformation function to this DataFrame.

pub fn random_split( &self, weights: Vec<f64>, seed: Option<i64>, ) -> Vec<DataFrame>

Randomly split this DataFrame into multiple parts.

pub fn print_schema(&self) -> Result<()>

Print the schema of this DataFrame.

pub fn storage_level(&self) -> Result<StorageLevel>

Get the storage level of this DataFrame. Mirrors DataFrame.storageLevel.

pub fn is_cached(&self) -> Result<bool>

Check if this DataFrame is cached. Mirrors DataFrame.is_cached.

Derived from the server-reported storage level (cached iff it uses memory or disk), rather than inspecting the local plan.

pub fn dtypes(&self) -> Result<Vec<(String, String)>>

Get dtypes (column names and types).

pub fn semantic_hash(&self) -> Result<i32>

Compute the server-side semantic hash of this DataFrame’s logical plan, mirroring DataFrame.semanticHash() (an AnalyzePlan request).

pub fn same_semantics(&self, other: &DataFrame) -> Result<bool>

Whether two DataFrames have the same semantics, mirroring DataFrame.sameSemantics(other) (a server-side AnalyzePlan comparison).

pub fn to_json(&self) -> Result<Vec<String>>

Convert each row to a JSON object string, mirroring DataFrame.toJSON().

Reference pyspark produces {"col":val,...} per row by applying the server’s to_json(struct(*)), not a client-side row rendering (which previously emitted Rust list syntax like [1, a]). Build that projection and collect the strings.

pub fn union_all(&self, other: &DataFrame) -> DataFrame

Union all rows (alias for union with all=true).

pub fn except_all(&self, other: &DataFrame) -> DataFrame

Except all rows.

pub fn intersect_all(&self, other: &DataFrame) -> DataFrame

Intersect all rows.

pub fn unpivot<C: Into<Column>, D: Into<Column>>( &self, ids: impl IntoIterator<Item = C>, values: Option<impl IntoIterator<Item = D>>, variable_column_name: &str, value_column_name: &str, ) -> DataFrame

Unpivot columns (like melt).

pub fn with_metadata( &self, column_name: &str, metadata: HashMap<String, String>, ) -> DataFrame

Set metadata on an existing column.

Mirrors pyspark.sql.connect.dataframe.DataFrame.withMetadata: the column is re-selected with the given metadata attached (serialized to a JSON map).

pub fn spark_session(&self) -> SparkSession

Get the Spark session.

pub fn is_local(&self) -> bool

Check if this DataFrame is local (collected).

pub fn is_streaming(&self) -> bool

Check if this DataFrame is streaming.

pub fn to_arrow(&self) -> Result<Vec<u8>>

Collect the DataFrame and serialize it to Arrow IPC (file format) bytes.

The returned buffer is a self-describing Arrow IPC stream that can be read back with arrow::ipc::reader::FileReader (or handed to pyarrow, polars, etc.). An empty result yields a valid IPC file with an empty schema.

pub fn repartition_by_id( &self, num_partitions: i32, partition_id_col: Column, ) -> DataFrame

Repartition into num_partitions using the given column’s value directly as the shuffle partition id. Mirrors DataFrame.repartitionById(numPartitions, partitionIdCol): the column is wrapped in a DirectShufflePartitionID expression and used as the sole repartition expression.

pub fn zip_with_index(&self, index_col_name: &str) -> DataFrame

Append a monotonically increasing index column. Mirrors DataFrame.zipWithIndex(indexColName="index"): self.select(col("*"), distributed_sequence_id().alias(indexColName)).

pub fn to(&self, schema: DataType) -> DataFrame

Reconcile this DataFrame to a new schema: reorder/select columns by name and cast them to the target types.

Mirrors pyspark.sql.connect.dataframe.DataFrame.to (a ToSchema relation).

pub fn exists(&self) -> Result<bool>

Check if the DataFrame exists (is not empty).

pub fn scalar(&self) -> Result<Option<Value>>

Get a scalar value from a single-row, single-column result.

pub fn transpose(&self) -> Result<DataFrame>

Transpose the DataFrame: swap rows and columns (server-side Transpose relation). Mirrors pyspark.sql.connect.dataframe.DataFrame.transpose() with no index column (the server uses the first column as the header).

pub fn transpose_with_index(&self, index_column: Column) -> Result<DataFrame>

Transpose using an explicit index column as the transposed header. Mirrors DataFrame.transpose(indexColumn).

pub fn zip(&self, other: &DataFrame) -> Result<DataFrame>

Zip this DataFrame with another DataFrame by row number.

pub fn register_temp_table(&self, name: &str) -> Result<()>

Register this DataFrame as a temporary table (deprecated - use createTempView).

pub fn as_table(&self, alias: &str) -> DataFrame

Convert to a table reference (alias for alias).

Trait Implementations

impl Clone for DataFrame

fn clone(&self) -> DataFrame

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.