robin-sparkless 4.13.1

PySpark-like DataFrame API in Rust on Polars; no JVM.
docs.rs failed to build robin-sparkless-4.13.1
Please check the build logs for more information.
See Builds for ideas on how to fix a failed build, or Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault, open an issue.
Visit the last successful build: robin-sparkless-4.4.0

Sparkless — PySpark-compatible DataFrames without the JVM

Install: pip install "sparkless>=4,<5"

Use Sparkless to run PySpark-style unit tests and local pipelines 10–100× faster in CI. Powered by the open-source Rust engine robin-sparkless (Polars execution).

Not a full cluster replacement. See Before you adopt for UDF, parity, and production caveats.

CI PyPI crates.io docs.rs Documentation License: MIT

Choose your path

I want to… Start here
Use Sparkless in Python (tests, local pipelines) python/README.md · pip install "sparkless>=4,<5"
Embed the Rust engine docs/QUICKSTART.md · robin-sparkless = "4" on crates.io
Contribute CONTRIBUTING.md · make check-full

Full documentation: Read the Docs

Quick start (Python)

# Swap the import—everything else stays the same.
from sparkless.sql import SparkSession, functions as F

spark = SparkSession.builder.app_name("demo").get_or_create()
df = spark.createDataFrame([{"x": 1}, {"x": 2}])
df.filter(F.col("x") > 1).show()

pip install "sparkless>=4,<5"

More: Python getting started · Testing guide · FAQ

Why Sparkless (Python)?

  • Familiar API — SparkSession, DataFrame, Column, and PySpark-like functions so you can reuse patterns without the JVM.
  • Fast local execution — Runs natively (no JVM) and uses Polars for IO, expressions, and aggregations.
  • Test the same suite two ways — Use sparkless.testing to run tests with Sparkless (fast) or real PySpark (parity checks).
  • Optional “Spark-like” features — SQL, temp/global temp views, saveAsTable, Delta, and JDBC (see python/README.md).

Features (Python surface)

Area What’s included
Core SparkSession, DataFrame, Column, functions
IO CSV, Parquet, JSON, Delta
Expressions col, lit, when/otherwise, casts, null handling
Aggregates count, sum, avg, min, max, groupBy().agg()
Window row_number, rank, dense_rank, lag, lead, first_value, last_value via .over()
Arrays, strings, JSON Common PySpark functions (explode, regexp_*, get_json_object, from_json, to_json, …)
SQL + views spark.sql, temp/global temp views, saveAsTable, catalog().listTables()
JDBC Read/write via spark.read.jdbc(...) / df.write.jdbc(...)

Parity: 200+ fixtures validated against PySpark. Before you adopt · PySpark differences · Parity status

Installation

Python (sparkless v4) — recommended

pip install "sparkless>=4,<5"

Contributors: pip install ./python or cd python && maturin develop — see CONTRIBUTING.md.

Rust engine (optional)

Most users should use the Python package above. To embed the engine in Rust:

[dependencies]
robin-sparkless = "4"

Optional features: sql, delta, jdbc, sqlite, jdbc_mysql. See docs/QUICKSTART.md.

Development

Prerequisites: Rust (see rust-toolchain.toml), Python 3.8+, maturin. Java only for SPARKLESS_TEST_MODE=pyspark.

See CONTRIBUTING.md for setup, make check-full, pytest, and maturin workflow.

Command Description
make check-full Full CI-equivalent check (Rust + Python)
pytest tests/ -v Python tests (sparkless backend)
SPARKLESS_TEST_MODE=pyspark pytest tests/ -v Same tests against real PySpark
make check Rust: format, clippy, audit, tests

Documentation

Resource Description
Python package Install, quick start, platform matrix, API overview
Read the Docs Getting started, testing, migration, FAQ
Before you adopt UDF limits, parity caveats, production notes
CONTRIBUTING Dev setup and PR checklist
docs.rs Rust API reference
Testing Guide Dual-mode testing with sparkless.testing
PySpark Differences Known divergences
RELEASING Publishing to crates.io and PyPI

See CHANGELOG.md for version history.

License

MIT