Crate datafusion_spark

Crate datafusion_spark 

Source
Expand description

Spark Expression packages for DataFusion.

This crate contains a collection of various Spark function packages for DataFusion, implemented using the extension API.

§Available Function Packages

See the list of modules in this crate for available packages.

§Example: using all function packages

You can register all the functions in all packages using the register_all function as shown below.

// Create a new session context
let mut ctx = SessionContext::new();
// register all spark functions with the context
datafusion_spark::register_all(&mut ctx)?;
// run a query. Note the `sha2` function is now available which
// has Spark semantics
let df = ctx.sql("SELECT sha2('The input String', 256)").await?;

§Example: calling a specific function in Rust

Each package also exports an expr_fn submodule that create Exprs for invoking functions via rust using a fluent style. For example, to invoke the sha2 function, you can use the following code:

use datafusion_spark::expr_fn::sha2;
// Create the expression `sha2(my_data, 256)`
let expr = sha2(col("my_data"), lit(256));

Modules§

expr_fn
Fluent-style API for creating Exprs
function

Functions§

all_default_aggregate_functions
Returns all default aggregate functions
all_default_scalar_functions
Returns all default scalar functions
all_default_table_functions
Returns all default table functions
all_default_window_functions
Returns all default window functions
register_all
Registers all enabled packages with a FunctionRegistry