Struct SparkSession 

pub struct SparkSession { /* private fields */ }

Main entry point for creating DataFrames and executing queries Similar to PySpark’s SparkSession but using Polars as the backend

Implementations

impl SparkSession

pub fn new( app_name: Option<String>, master: Option<String>, config: HashMap<String, String>, ) -> Self

pub fn create_or_replace_temp_view(&self, name: &str, df: DataFrame)

Register a DataFrame as a temporary view (PySpark: createOrReplaceTempView). The view is session-scoped and is dropped when the session is dropped.

pub fn create_global_temp_view(&self, name: &str, df: DataFrame)

Global temp view (PySpark: createGlobalTempView). Stub: uses same catalog as temp view.

pub fn create_or_replace_global_temp_view(&self, name: &str, df: DataFrame)

Global temp view (PySpark: createOrReplaceGlobalTempView). Stub: uses same catalog as temp view.

pub fn drop_temp_view(&self, name: &str)

Drop a temporary view by name (PySpark: catalog.dropTempView). No error if the view does not exist.

pub fn drop_global_temp_view(&self, name: &str)

Drop a global temporary view (PySpark: catalog.dropGlobalTempView). Stub: same catalog as temp view.

pub fn table_exists(&self, name: &str) -> bool

Check if a temporary view exists.

pub fn list_temp_view_names(&self) -> Vec<String>

Return temporary view names in this session.

pub fn table(&self, name: &str) -> Result<DataFrame, PolarsError>

Look up a temporary view by name (PySpark: table(name)). Returns an error if the view does not exist.

pub fn builder() -> SparkSessionBuilder

pub fn get_config(&self) -> &HashMap<String, String>

Return a reference to the session config (for catalog/conf compatibility).

pub fn is_case_sensitive(&self) -> bool

Whether column names are case-sensitive (PySpark: spark.sql.caseSensitive). Default is false (case-insensitive matching).

pub fn create_dataframe( &self, data: Vec<(i64, i64, String)>, column_names: Vec<&str>, ) -> Result<DataFrame, PolarsError>

Create a DataFrame from a vector of tuples (i64, i64, String)

Example

use robin_sparkless::session::SparkSession;

let spark = SparkSession::builder().app_name("test").get_or_create();
let df = spark.create_dataframe(
    vec![
        (1, 25, "Alice".to_string()),
        (2, 30, "Bob".to_string()),
    ],
    vec!["id", "age", "name"],
)?;

pub fn create_dataframe_from_polars(&self, df: PlDataFrame) -> DataFrame

Create a DataFrame from a Polars DataFrame

pub fn create_dataframe_from_rows( &self, rows: Vec<Vec<JsonValue>>, schema: Vec<(String, String)>, ) -> Result<DataFrame, PolarsError>

Create a DataFrame from rows and a schema (arbitrary column count and types).

rows: each inner vec is one row; length must match schema length. Values are JSON-like (i64, f64, string, bool, null). schema: list of (column_name, dtype_string), e.g. [("id", "bigint"), ("name", "string")]. Supported dtype strings: bigint, int, long, double, float, string, str, varchar, boolean, bool, date, timestamp, datetime.

pub fn range( &self, start: i64, end: i64, step: i64, ) -> Result<DataFrame, PolarsError>

Create a DataFrame with a single column id (bigint) containing values from start to end (exclusive) with step. PySpark: spark.range(end) or spark.range(start, end, step).

• range(end) → 0 to end-1, step 1
• range(start, end) → start to end-1, step 1
• range(start, end, step) → start, start+step, … up to but not including end

pub fn read_csv(&self, path: impl AsRef<Path>) -> Result<DataFrame, PolarsError>

Read a CSV file.

Uses Polars’ CSV reader with default options:

• Header row is inferred (default: true)
• Schema is inferred from first 100 rows

Example

use robin_sparkless::SparkSession;

let spark = SparkSession::builder().app_name("test").get_or_create();
let df_result = spark.read_csv("data.csv");
// Handle the Result as appropriate in your application

pub fn read_parquet( &self, path: impl AsRef<Path>, ) -> Result<DataFrame, PolarsError>

Read a Parquet file.

Uses Polars’ Parquet reader. Parquet files have embedded schema, so schema inference is automatic.

Example

use robin_sparkless::SparkSession;

let spark = SparkSession::builder().app_name("test").get_or_create();
let df_result = spark.read_parquet("data.parquet");
// Handle the Result as appropriate in your application

pub fn read_json( &self, path: impl AsRef<Path>, ) -> Result<DataFrame, PolarsError>

Read a JSON file (JSONL format - one JSON object per line).

Uses Polars’ JSONL reader with default options:

• Schema is inferred from first 100 rows

Example

use robin_sparkless::SparkSession;

let spark = SparkSession::builder().app_name("test").get_or_create();
let df_result = spark.read_json("data.json");
// Handle the Result as appropriate in your application

pub fn sql(&self, _query: &str) -> Result<DataFrame, PolarsError>

Execute a SQL query (stub when sql feature is disabled).

pub fn read_delta( &self, _path: impl AsRef<Path>, ) -> Result<DataFrame, PolarsError>

Stub when delta feature is disabled.

pub fn read_delta_with_version( &self, _path: impl AsRef<Path>, _version: Option<i64>, ) -> Result<DataFrame, PolarsError>

pub fn stop(&self)

Stop the session (cleanup resources)

impl SparkSession

pub fn read(&self) -> DataFrameReader

Get a DataFrameReader for reading files

Trait Implementations

impl Clone for SparkSession

fn clone(&self) -> SparkSession

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for SparkSession

fn default() -> Self

Returns the “default value” for a type.