Expand description
Spark Expression packages for DataFusion.
This crate contains a collection of various Spark function packages for DataFusion, implemented using the extension API.
§Available Function Packages
See the list of modules in this crate for available packages.
§Example: using all function packages
You can register all the functions in all packages using the register_all
function as shown below.
// Create a new session context
let mut ctx = SessionContext::new();
// register all spark functions with the context
datafusion_spark::register_all(&mut ctx)?;
// run a query. Note the `sha2` function is now available which
// has Spark semantics
let df = ctx.sql("SELECT sha2('The input String', 256)").await?;
§Example: calling a specific function in Rust
Each package also exports an expr_fn
submodule that create Expr
s for
invoking functions via rust using a fluent style. For example, to invoke the
sha2
function, you can use the following code:
use datafusion_spark::expr_fn::sha2;
// Create the expression `sha2(my_data, 256)`
let expr = sha2(col("my_data"), lit(256));
Modules§
Functions§
- all_
default_ aggregate_ functions - Returns all default aggregate functions
- all_
default_ scalar_ functions - Returns all default scalar functions
- all_
default_ table_ functions - Returns all default table functions
- all_
default_ window_ functions - Returns all default window functions
- register_
all - Registers all enabled packages with a
FunctionRegistry