Enum datafusion_expr::expr::Expr
source · pub enum Expr {
Show 34 variants
Alias(Alias),
Column(Column),
ScalarVariable(DataType, Vec<String>),
Literal(ScalarValue),
BinaryExpr(BinaryExpr),
Like(Like),
SimilarTo(Like),
Not(Box<Expr>),
IsNotNull(Box<Expr>),
IsNull(Box<Expr>),
IsTrue(Box<Expr>),
IsFalse(Box<Expr>),
IsUnknown(Box<Expr>),
IsNotTrue(Box<Expr>),
IsNotFalse(Box<Expr>),
IsNotUnknown(Box<Expr>),
Negative(Box<Expr>),
Between(Between),
Case(Case),
Cast(Cast),
TryCast(TryCast),
Sort(Sort),
ScalarFunction(ScalarFunction),
AggregateFunction(AggregateFunction),
WindowFunction(WindowFunction),
InList(InList),
Exists(Exists),
InSubquery(InSubquery),
ScalarSubquery(Subquery),
Wildcard {
qualifier: Option<String>,
},
GroupingSet(GroupingSet),
Placeholder(Placeholder),
OuterReferenceColumn(DataType, Column),
Unnest(Unnest),
}
Expand description
Represents logical expressions such as A + 1
, or CAST(c1 AS int)
.
For example the expression A + 1
will be represented as
BinaryExpr {
left: Expr::Column("A"),
op: Operator::Plus,
right: Expr::Literal(ScalarValue::Int32(Some(1)))
}
§Creating Expressions
Expr
s can be created directly, but it is often easier and less verbose to
use the fluent APIs in crate::expr_fn
such as col
and lit
, or
methods such as Expr::alias
, Expr::cast_to
, and Expr::Like
).
§Schema Access
See ExprSchemable::get_type
to access the DataType
and nullability
of an Expr
.
§Visiting and Rewriting Expr
s
The Expr
struct implements the TreeNode
trait for walking and
rewriting expressions. For example TreeNode::apply
recursively visits an
Expr
and TreeNode::transform
can be used to rewrite an expression. See
the examples below and TreeNode
for more information.
§Examples
§Column references and literals
Expr::Column
refer to the values of columns and are often created with
the col
function. For example to create an expression c1
referring to
column named “c1”:
let expr = col("c1");
assert_eq!(expr, Expr::Column(Column::from_name("c1")));
Expr::Literal
refer to literal, or constant, values. These are created
with the lit
function. For example to create an expression 42
:
// All literals are strongly typed in DataFusion. To make an `i64` 42:
let expr = lit(42i64);
assert_eq!(expr, Expr::Literal(ScalarValue::Int64(Some(42))));
// To make a (typed) NULL:
let expr = Expr::Literal(ScalarValue::Int64(None));
// to make an (untyped) NULL (the optimizer will coerce this to the correct type):
let expr = lit(ScalarValue::Null);
§Binary Expressions
Exprs implement traits that allow easy to understand construction of more
complex expresions. For example, to create c1 + c2
to add columns “c1” and
“c2” together
// Use the `+` operator to add two columns together
let expr = col("c1") + col("c2");
assert!(matches!(expr, Expr::BinaryExpr { ..} ));
if let Expr::BinaryExpr(binary_expr) = expr {
assert_eq!(*binary_expr.left, col("c1"));
assert_eq!(*binary_expr.right, col("c2"));
assert_eq!(binary_expr.op, Operator::Plus);
}
The expression c1 = 42
to compares the value in column “c1” to the
literal value 42
:
let expr = col("c1").eq(lit(42_i32));
assert!(matches!(expr, Expr::BinaryExpr { .. } ));
if let Expr::BinaryExpr(binary_expr) = expr {
assert_eq!(*binary_expr.left, col("c1"));
let scalar = ScalarValue::Int32(Some(42));
assert_eq!(*binary_expr.right, Expr::Literal(scalar));
assert_eq!(binary_expr.op, Operator::Eq);
}
Here is how to implement the equivalent of SELECT *
to select all
Expr::Column
from a DFSchema
’s columns:
// Create a schema c1(int, c2 float)
let arrow_schema = Schema::new(vec![
Field::new("c1", DataType::Int32, false),
Field::new("c2", DataType::Float64, false),
]);
// DFSchema is a an Arrow schema with optional relation name
let df_schema = DFSchema::try_from_qualified_schema("t1", &arrow_schema)
.unwrap();
// Form Vec<Expr> with an expression for each column in the schema
let exprs: Vec<_> = df_schema.iter()
.map(Expr::from)
.collect();
assert_eq!(exprs, vec![
Expr::from(Column::from_qualified_name("t1.c1")),
Expr::from(Column::from_qualified_name("t1.c2")),
]);
§Visiting and Rewriting Expr
s
Here is an example that finds all literals in an Expr
tree:
use datafusion_common::ScalarValue;
use datafusion_common::tree_node::{TreeNode, TreeNodeRecursion};
// Expression a = 5 AND b = 6
let expr = col("a").eq(lit(5)) & col("b").eq(lit(6));
// find all literals in a HashMap
let mut scalars = HashSet::new();
// apply recursively visits all nodes in the expression tree
expr.apply(|e| {
if let Expr::Literal(scalar) = e {
scalars.insert(scalar);
}
// The return value controls whether to continue visiting the tree
Ok(TreeNodeRecursion::Continue)
}).unwrap();;
// All subtrees have been visited and literals found
assert_eq!(scalars.len(), 2);
assert!(scalars.contains(&ScalarValue::Int32(Some(5))));
assert!(scalars.contains(&ScalarValue::Int32(Some(6))));
Rewrite an expression, replacing references to column “a” in an
to the literal 42
:
// expression a = 5 AND b = 6
let expr = col("a").eq(lit(5)).and(col("b").eq(lit(6)));
// rewrite all references to column "a" to the literal 42
let rewritten = expr.transform(|e| {
if let Expr::Column(c) = &e {
if &c.name == "a" {
// return Transformed::yes to indicate the node was changed
return Ok(Transformed::yes(lit(42)))
}
}
// return Transformed::no to indicate the node was not changed
Ok(Transformed::no(e))
}).unwrap();
// The expression has been rewritten
assert!(rewritten.transformed);
// to 42 = 5 AND b = 6
assert_eq!(rewritten.data, lit(42).eq(lit(5)).and(col("b").eq(lit(6))));
Variants§
Alias(Alias)
An expression with a specific name.
Column(Column)
A named reference to a qualified filed in a schema.
ScalarVariable(DataType, Vec<String>)
A named reference to a variable in a registry.
Literal(ScalarValue)
A constant value.
BinaryExpr(BinaryExpr)
A binary expression such as “age > 21”
Like(Like)
LIKE expression
SimilarTo(Like)
LIKE expression that uses regular expressions
Not(Box<Expr>)
Negation of an expression. The expression’s type must be a boolean to make sense.
IsNotNull(Box<Expr>)
True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>)
True if argument is NULL, false otherwise. This expression itself is never NULL.
IsTrue(Box<Expr>)
True if argument is true, false otherwise. This expression itself is never NULL.
IsFalse(Box<Expr>)
True if argument is false, false otherwise. This expression itself is never NULL.
IsUnknown(Box<Expr>)
True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNotTrue(Box<Expr>)
True if argument is FALSE or NULL, false otherwise. This expression itself is never NULL.
IsNotFalse(Box<Expr>)
True if argument is TRUE OR NULL, false otherwise. This expression itself is never NULL.
IsNotUnknown(Box<Expr>)
True if argument is TRUE or FALSE, false otherwise. This expression itself is never NULL.
Negative(Box<Expr>)
arithmetic negation of an expression, the operand must be of a signed numeric data type
Between(Between)
Whether an expression is between a given range.
Case(Case)
The CASE expression is similar to a series of nested if/else and there are two forms that can be used. The first form consists of a series of boolean “when” expressions with corresponding “then” expressions, and an optional “else” expression.
CASE WHEN condition THEN result [WHEN …] [ELSE result] END
The second form uses a base expression and then a series of “when” clauses that match on a literal value.
CASE expression WHEN value THEN result [WHEN …] [ELSE result] END
Cast(Cast)
Casts the expression to a given type and will return a runtime error if the expression cannot be cast. This expression is guaranteed to have a fixed type.
TryCast(TryCast)
Casts the expression to a given type and will return a null value if the expression cannot be cast. This expression is guaranteed to have a fixed type.
Sort(Sort)
A sort expression, that can be used to sort values.
ScalarFunction(ScalarFunction)
Represents the call of a scalar function with a set of arguments.
AggregateFunction(AggregateFunction)
Represents the call of an aggregate built-in function with arguments.
WindowFunction(WindowFunction)
Represents the call of a window function with arguments.
InList(InList)
Returns whether the list contains the expr value.
Exists(Exists)
EXISTS subquery
InSubquery(InSubquery)
IN subquery
ScalarSubquery(Subquery)
Scalar subquery
Wildcard
Represents a reference to all available fields in a specific schema, with an optional (schema) qualifier.
This expr has to be resolved to a list of columns before translating logical plan into physical plan.
GroupingSet(GroupingSet)
List of grouping set expressions. Only valid in the context of an aggregate GROUP BY expression list
Placeholder(Placeholder)
A place holder for parameters in a prepared statement
(e.g. $foo
or $1
)
OuterReferenceColumn(DataType, Column)
A place holder which hold a reference to a qualified field in the outer query, used for correlated sub queries.
Unnest(Unnest)
Unnest expression
Implementations§
source§impl Expr
impl Expr
sourcepub fn display_name(&self) -> Result<String>
pub fn display_name(&self) -> Result<String>
Returns the name of this expression as it should appear in a schema. This name will not include any CAST expressions.
sourcepub fn canonical_name(&self) -> String
pub fn canonical_name(&self) -> String
Returns a full and complete string representation of this expression.
sourcepub fn variant_name(&self) -> &str
pub fn variant_name(&self) -> &str
Return String representation of the variant represented by self
Useful for non-rust based bindings
sourcepub fn name_for_alias(&self) -> Result<String>
pub fn name_for_alias(&self) -> Result<String>
Return the name to use for the specific Expr, recursing into
Expr::Sort
as appropriate
sourcepub fn alias_if_changed(self, original_name: String) -> Result<Expr>
pub fn alias_if_changed(self, original_name: String) -> Result<Expr>
Ensure expr
has the name as original_name
by adding an
alias if necessary.
sourcepub fn alias_qualified(
self,
relation: Option<impl Into<TableReference>>,
name: impl Into<String>,
) -> Expr
pub fn alias_qualified( self, relation: Option<impl Into<TableReference>>, name: impl Into<String>, ) -> Expr
Return self AS name
alias expression with a specific qualifier
sourcepub fn unalias(self) -> Expr
pub fn unalias(self) -> Expr
Remove an alias from an expression if one exists.
If the expression is not an alias, the expression is returned unchanged. This method does not remove aliases from nested expressions.
§Example
// `foo as "bar"` is unaliased to `foo`
let expr = col("foo").alias("bar");
assert_eq!(expr.unalias(), col("foo"));
// `foo as "bar" + baz` is not unaliased
let expr = col("foo").alias("bar") + col("baz");
assert_eq!(expr.clone().unalias(), expr);
// `foo as "bar" as "baz" is unalaised to foo as "bar"
let expr = col("foo").alias("bar").alias("baz");
assert_eq!(expr.unalias(), col("foo").alias("bar"));
sourcepub fn unalias_nested(self) -> Expr
pub fn unalias_nested(self) -> Expr
Recursively potentially multiple aliases from an expression.
If the expression is not an alias, the expression is returned unchanged. This method removes directly nested aliases, but not other nested aliases.
§Example
// `foo as "bar"` is unaliased to `foo`
let expr = col("foo").alias("bar");
assert_eq!(expr.unalias_nested(), col("foo"));
// `foo as "bar" + baz` is not unaliased
let expr = col("foo").alias("bar") + col("baz");
assert_eq!(expr.clone().unalias_nested(), expr);
// `foo as "bar" as "baz" is unalaised to foo
let expr = col("foo").alias("bar").alias("baz");
assert_eq!(expr.unalias_nested(), col("foo"));
sourcepub fn in_list(self, list: Vec<Expr>, negated: bool) -> Expr
pub fn in_list(self, list: Vec<Expr>, negated: bool) -> Expr
Return self IN <list>
if negated
is false, otherwise
return self NOT IN <list>
.a
sourcepub fn is_not_null(self) -> Expr
pub fn is_not_null(self) -> Expr
Return `IsNotNull(Box(self))
sourcepub fn sort(self, asc: bool, nulls_first: bool) -> Expr
pub fn sort(self, asc: bool, nulls_first: bool) -> Expr
Create a sort expression from an existing expression.
let sort_expr = col("foo").sort(true, true); // SORT ASC NULLS_FIRST
sourcepub fn is_not_true(self) -> Expr
pub fn is_not_true(self) -> Expr
Return IsNotTrue(Box(self))
sourcepub fn is_not_false(self) -> Expr
pub fn is_not_false(self) -> Expr
Return IsNotFalse(Box(self))
sourcepub fn is_unknown(self) -> Expr
pub fn is_unknown(self) -> Expr
Return IsUnknown(Box(self))
sourcepub fn is_not_unknown(self) -> Expr
pub fn is_not_unknown(self) -> Expr
Return IsNotUnknown(Box(self))
sourcepub fn not_between(self, low: Expr, high: Expr) -> Expr
pub fn not_between(self, low: Expr, high: Expr) -> Expr
return self NOT BETWEEN low AND high
pub fn try_into_col(&self) -> Result<Column>
sourcepub fn try_as_col(&self) -> Option<&Column>
pub fn try_as_col(&self) -> Option<&Column>
Return a reference to the inner Column
if any
returns None
if the expression is not a Column
Example
use datafusion_expr::{col, Expr};
let expr = col("foo");
assert_eq!(expr.try_as_col(), Some(&Column::from("foo")));
let expr = col("foo").alias("bar");
assert_eq!(expr.try_as_col(), None);
sourcepub fn to_columns(&self) -> Result<HashSet<Column>>
pub fn to_columns(&self) -> Result<HashSet<Column>>
Return all referenced columns of this expression.
sourcepub fn contains_outer(&self) -> bool
pub fn contains_outer(&self) -> bool
Return true when the expression contains out reference(correlated) expressions.
sourcepub fn is_volatile(&self) -> Result<bool>
pub fn is_volatile(&self) -> Result<bool>
Returns true if the expression is volatile, i.e. whether it can return different results when evaluated multiple times with the same input.
sourcepub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<Expr>
pub fn infer_placeholder_types(self, schema: &DFSchema) -> Result<Expr>
Recursively find all Expr::Placeholder
expressions, and
to infer their DataType
from the context of their use.
For example, gicen an expression like <int32> = $0
will infer $0
to
have type int32
.
sourcepub fn short_circuits(&self) -> bool
pub fn short_circuits(&self) -> bool
Returns true if some of this exprs
subexpressions may not be evaluated
and thus any side effects (like divide by zero) may not be encountered
Trait Implementations§
source§impl Display for Expr
impl Display for Expr
Format expressions for display as part of a logical plan. In many cases, this will produce
similar output to Expr.name()
except that column names will be prefixed with ‘#’.
source§impl ExprSchemable for Expr
impl ExprSchemable for Expr
source§fn get_type(&self, schema: &dyn ExprSchema) -> Result<DataType>
fn get_type(&self, schema: &dyn ExprSchema) -> Result<DataType>
Returns the arrow::datatypes::DataType of the expression based on ExprSchema
Note: DFSchema
implements ExprSchema.
§Examples
Get the type of an expression that adds 2 columns. Adding an Int32 and Float32 results in Float32 type
fn main() {
let expr = col("c1") + col("c2");
let schema = DFSchema::from_unqualifed_fields(
vec![
Field::new("c1", DataType::Int32, true),
Field::new("c2", DataType::Float32, true),
].into(),
HashMap::new(),
).unwrap();
assert_eq!("Float32", format!("{}", expr.get_type(&schema).unwrap()));
}
§Errors
This function errors when it is not possible to compute its
arrow::datatypes::DataType. This happens when e.g. the
expression refers to a column that does not exist in the
schema, or when the expression is incorrectly typed
(e.g. [utf8] + [bool]
).
source§fn nullable(&self, input_schema: &dyn ExprSchema) -> Result<bool>
fn nullable(&self, input_schema: &dyn ExprSchema) -> Result<bool>
Returns the nullability of the expression based on ExprSchema.
Note: DFSchema
implements ExprSchema.
§Errors
This function errors when it is not possible to compute its nullability. This happens when the expression refers to a column that does not exist in the schema.
source§fn data_type_and_nullable(
&self,
schema: &dyn ExprSchema,
) -> Result<(DataType, bool)>
fn data_type_and_nullable( &self, schema: &dyn ExprSchema, ) -> Result<(DataType, bool)>
Returns the datatype and nullability of the expression based on ExprSchema.
Note: DFSchema
implements ExprSchema.
§Errors
This function errors when it is not possible to compute its datatype or nullability.
source§fn to_field(
&self,
input_schema: &dyn ExprSchema,
) -> Result<(Option<TableReference>, Arc<Field>)>
fn to_field( &self, input_schema: &dyn ExprSchema, ) -> Result<(Option<TableReference>, Arc<Field>)>
Returns a arrow::datatypes::Field compatible with this expression.
So for example, a projected expression col(c1) + col(c2)
is
placed in an output field named col(“c1 + c2”)
source§fn cast_to(
self,
cast_to_type: &DataType,
schema: &dyn ExprSchema,
) -> Result<Expr>
fn cast_to( self, cast_to_type: &DataType, schema: &dyn ExprSchema, ) -> Result<Expr>
Wraps this expression in a cast to a target arrow::datatypes::DataType.
§Errors
This function errors when it is impossible to cast the expression to the target arrow::datatypes::DataType.
source§impl<'a> From<(Option<&'a TableReference>, &'a Arc<Field>)> for Expr
impl<'a> From<(Option<&'a TableReference>, &'a Arc<Field>)> for Expr
source§impl PartialEq for Expr
impl PartialEq for Expr
source§impl PartialOrd for Expr
impl PartialOrd for Expr
1.0.0 · source§fn le(&self, other: &Rhs) -> bool
fn le(&self, other: &Rhs) -> bool
self
and other
) and is used by the <=
operator. Read moresource§impl TreeNode for Expr
impl TreeNode for Expr
source§fn apply_children<'n, F: FnMut(&'n Self) -> Result<TreeNodeRecursion>>(
&'n self,
f: F,
) -> Result<TreeNodeRecursion>
fn apply_children<'n, F: FnMut(&'n Self) -> Result<TreeNodeRecursion>>( &'n self, f: F, ) -> Result<TreeNodeRecursion>
source§fn map_children<F: FnMut(Self) -> Result<Transformed<Self>>>(
self,
f: F,
) -> Result<Transformed<Self>>
fn map_children<F: FnMut(Self) -> Result<Transformed<Self>>>( self, f: F, ) -> Result<Transformed<Self>>
source§fn visit<'n, V>(
&'n self,
visitor: &mut V,
) -> Result<TreeNodeRecursion, DataFusionError>where
V: TreeNodeVisitor<'n, Node = Self>,
fn visit<'n, V>(
&'n self,
visitor: &mut V,
) -> Result<TreeNodeRecursion, DataFusionError>where
V: TreeNodeVisitor<'n, Node = Self>,
TreeNodeVisitor
, performing a
depth-first walk of the node and its children. Read moresource§fn rewrite<R>(
self,
rewriter: &mut R,
) -> Result<Transformed<Self>, DataFusionError>where
R: TreeNodeRewriter<Node = Self>,
fn rewrite<R>(
self,
rewriter: &mut R,
) -> Result<Transformed<Self>, DataFusionError>where
R: TreeNodeRewriter<Node = Self>,
TreeNodeRewriter
, performing a
depth-first walk of the node and its children. Read moresource§fn apply<'n, F>(&'n self, f: F) -> Result<TreeNodeRecursion, DataFusionError>
fn apply<'n, F>(&'n self, f: F) -> Result<TreeNodeRecursion, DataFusionError>
f
to the node then each of its children, recursively (a
top-down, pre-order traversal). Read moresource§fn transform<F>(self, f: F) -> Result<Transformed<Self>, DataFusionError>
fn transform<F>(self, f: F) -> Result<Transformed<Self>, DataFusionError>
f
(a bottom-up post-order traversal). Read moresource§fn transform_down<F>(self, f: F) -> Result<Transformed<Self>, DataFusionError>
fn transform_down<F>(self, f: F) -> Result<Transformed<Self>, DataFusionError>
f
in a top-down (pre-order)
fashion. Read moresource§fn transform_down_mut<F>(
self,
f: &mut F,
) -> Result<Transformed<Self>, DataFusionError>
fn transform_down_mut<F>( self, f: &mut F, ) -> Result<Transformed<Self>, DataFusionError>
transform_down
insteadSelf::transform_down
but with a mutable closure.source§fn transform_up<F>(self, f: F) -> Result<Transformed<Self>, DataFusionError>
fn transform_up<F>(self, f: F) -> Result<Transformed<Self>, DataFusionError>
f
in a bottom-up (post-order)
fashion. Read moresource§fn transform_up_mut<F>(
self,
f: &mut F,
) -> Result<Transformed<Self>, DataFusionError>
fn transform_up_mut<F>( self, f: &mut F, ) -> Result<Transformed<Self>, DataFusionError>
transform_up
insteadSelf::transform_up
but with a mutable closure.source§fn transform_down_up<FD, FU>(
self,
f_down: FD,
f_up: FU,
) -> Result<Transformed<Self>, DataFusionError>where
FD: FnMut(Self) -> Result<Transformed<Self>, DataFusionError>,
FU: FnMut(Self) -> Result<Transformed<Self>, DataFusionError>,
fn transform_down_up<FD, FU>(
self,
f_down: FD,
f_up: FU,
) -> Result<Transformed<Self>, DataFusionError>where
FD: FnMut(Self) -> Result<Transformed<Self>, DataFusionError>,
FU: FnMut(Self) -> Result<Transformed<Self>, DataFusionError>,
f_down
while traversing the tree top-down
(pre-order), and using f_up
while traversing the tree bottom-up
(post-order). Read moreimpl Eq for Expr
impl StructuralPartialEq for Expr
Auto Trait Implementations§
impl Freeze for Expr
impl !RefUnwindSafe for Expr
impl Send for Expr
impl Sync for Expr
impl Unpin for Expr
impl !UnwindSafe for Expr
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
source§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
source§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
source§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
key
and return true
if they are equal.