Crate datafusion_spark 

Spark Expression packages for DataFusion.

This crate contains a collection of various Spark function packages for DataFusion, implemented using the extension API.

Available Function Packages

See the list of modules in this crate for available packages.

Example: using all function packages

You can register all the functions in all packages using the register_all function as shown below. Any existing functions will be overwritten, with these Spark functions taking priority.

// Create a new session context
let mut ctx = SessionContext::new();
// Register all Spark functions with the context
datafusion_spark::register_all(&mut ctx)?;
// Run a query using the `sha2` function which is now available and has Spark semantics
let df = ctx.sql("SELECT sha2('The input String', 256)").await?;

Example: calling a specific function in Rust

Each package also exports an expr_fn submodule that create Exprs for invoking functions via rust using a fluent style. For example, to invoke the sha2 function, you can use the following code:

use datafusion_spark::expr_fn::sha2;
// Create the expression `sha2(my_data, 256)`
let expr = sha2(col("my_data"), lit(256));

Modules

expr_fn

Fluent-style API for creating Exprs

function

Functions

all_default_aggregate_functions

Returns all default aggregate functions

all_default_scalar_functions

Returns all default scalar functions

all_default_table_functions

Returns all default table functions

all_default_window_functions

Returns all default window functions

register_all

Registers all enabled packages with a FunctionRegistry, overriding any existing functions if there is a name clash.