Module deltalake::datafusion::prelude

DataFusion “prelude” to simplify importing common types.

Like the standard library’s prelude, this module simplifies importing of common items. Unlike the standard prelude, the contents of this module must be imported manually:

use datafusion::prelude::*;

Structs

• AvroReadOptions
  Options that control the reading of AVRO files.
• Column
  A named reference to a qualified field in a schema.
• CsvReadOptions
  Options that control the reading of CSV files.
• DataFrame
  DataFrame represents a logical set of rows with the same named columns. Similar to a Pandas DataFrame or Spark DataFrame
• NdJsonReadOptions
  Options that control the reading of Line-delimited JSON files (NDJson)
• ParquetReadOptions
  Options that control the reading of Parquet files.
• SQLOptions
  Describes which SQL statements can be run.
• SessionConfig
  Configuration options for SessionContext.
• SessionContext
  Main interface for executing queries with DataFusion. Maintains the state of the connection between a user and an instance of the DataFusion engine.
• SimpleAggregateUDF
  Implements AggregateUDFImpl for functions that have a single signature and return type.
• SimpleScalarUDF
  Implements ScalarUDFImpl for functions that have a single signature and return type.
• SimpleWindowUDF
  Implements WindowUDFImpl for functions that have a single signature and return type.

Enums

• Expr
  Expr is a central struct of DataFusion’s query API, and represent logical expressions such as A + 1, or CAST(c1 AS int).
• JoinType
  Join type
• Partitioning
  Logical partitioning schemes supported by the repartition operator.

Traits

• Add
  The addition operator +.
• BitAnd
  The bitwise AND operator &.
• BitOr
  The bitwise OR operator |.
• BitXor
  The bitwise XOR operator ^.
• Div
  The division operator /.
• Mul
  The multiplication operator *.
• Neg
  The unary negation operator -.
• Not
  The unary logical negation operator !.
• Rem
  The remainder operator %.
• Shl
  The left shift operator <<. Note that because this trait is implemented for all integer types with multiple right-hand-side types, Rust’s type checker has special handling for _ << _, setting the result type for integer operations to the type of the left-hand-side operand. This means that though a << b and a.shl(b) are one and the same from an evaluation standpoint, they are different when it comes to type inference.
• Shr
  The right shift operator >>. Note that because this trait is implemented for all integer types with multiple right-hand-side types, Rust’s type checker has special handling for _ >> _, setting the result type for integer operations to the type of the left-hand-side operand. This means that though a >> b and a.shr(b) are one and the same from an evaluation standpoint, they are different when it comes to type inference.
• Sub
  The subtraction operator -.

Functions

• abs
  absolute value
• acos
  inverse cosine
• acosh
  inverse hyperbolic cosine
• and
  Return a new expression with a logical AND
• approx_distinct
  Returns the approximate number of distinct input values. This function provides an approximation of count(DISTINCT x). Zero is returned if all input values are null. This function should produce a standard error of 0.81%, which is the standard deviation of the (approximately normal) error distribution over all possible sets. It does not guarantee an upper bound on the error for any specific input set.
• approx_median
  Calculate an approximation of the median for expr.
• approx_percentile_cont
  Calculate an approximation of the specified percentile for expr.
• approx_percentile_cont_with_weight
  Calculate an approximation of the specified percentile for expr and weight_expr.
• array
  returns an Arrow array using the specified input expressions.
• array_agg
  Create an expression to represent the array_agg() aggregate function
• array_append
  appends an element to the end of an array.
• array_concat
  concatenates arrays.
• array_dims
  returns an array of the array’s dimensions.
• array_distinct
  return distinct values from the array after removing duplicates.
• array_element
  extracts the element with the index n from the array.
• array_empty
  returns 1 for an empty array or 0 for a non-empty array.
• array_except
  Returns an array of the elements that appear in the first array but not in the second.
• array_has
  returns true, if the element appears in the first array, otherwise false.
• array_has_all
  returns true if each element of the second array appears in the first array; otherwise, it returns false.
• array_has_any
  returns true if at least one element of the second array appears in the first array; otherwise, it returns false.
• array_intersect
  Returns an array of the elements in the intersection of array1 and array2.
• array_length
  returns the length of the array dimension.
• array_ndims
  returns the number of dimensions of the array.
• array_pop_back
  returns the array without the last element.
• array_pop_front
  returns the array without the first element.
• array_position
  searches for an element in the array, returns first occurrence.
• array_positions
  searches for an element in the array, returns all occurrences.
• array_prepend
  prepends an element to the beginning of an array.
• array_remove
  removes the first element from the array equal to the given value.
• array_remove_all
  removes all elements from the array equal to the given value.
• array_remove_n
  removes the first max elements from the array equal to the given value.
• array_repeat
  returns an array containing element count times.
• array_replace
  replaces the first occurrence of the specified element with another specified element.
• array_replace_all
  replaces all occurrences of the specified element with another specified element.
• array_replace_n
  replaces the first max occurrences of the specified element with another specified element.
• array_resize
  returns an array with the specified size filled with the given value.
• array_slice
  returns a slice of the array.
• array_sort
  returns sorted array.
• array_to_string
  converts each element to its text representation.
• array_union
  returns an array of the elements in the union of array1 and array2 without duplicates.
• arrow_typeof
  data type
• ascii
  ASCII code value of the character
• asin
  inverse sine
• asinh
  inverse hyperbolic sine
• atan
  inverse tangent
• atan2
  inverse tangent of a division given in the argument
• atanh
  inverse hyperbolic tangent
• avg
  Create an expression to represent the avg() aggregate function
• binary_expr
  Return a new expression left <op> right
• bit_length
  the number of bits in the string
• bitwise_and
  Return a new expression with bitwise AND
• bitwise_or
  Return a new expression with bitwise OR
• bitwise_shift_left
  Return a new expression with bitwise SHIFT LEFT
• bitwise_shift_right
  Return a new expression with bitwise SHIFT RIGHT
• bitwise_xor
  Return a new expression with bitwise XOR
• btrim
  removes all characters, spaces by default, from both sides of a string
• call_fn
  Calls a named built in function
• cardinality
  returns the total number of elements in the array.
• case
  Create a CASE WHEN statement with literal WHEN expressions for comparison to the base expression.
• cast
  Create a cast expression
• cbrt
  cube root of a number
• ceil
  nearest integer greater than or equal to argument
• character_length
  the number of characters in the string
• chr
  converts the Unicode code point to a UTF8 character
• coalesce
  returns coalesce(args...), which evaluates to the value of the first Expr which is not NULL
• col
  Create a column expression based on a qualified or unqualified column name. Will normalize unquoted identifiers according to SQL rules (identifiers will become lowercase).
• concat
  Concatenates the text representations of all the arguments. NULL arguments are ignored.
• concat_expr
  concatenates several strings
• concat_ws
  Concatenates all but the first argument, with separators. The first argument is used as the separator. NULL arguments in values are ignored.
• concat_ws_expr
  concatenates several strings, placing a seperator between each one
• cos
  cosine
• cosh
  hyperbolic cosine
• cot
  cotangent
• count
  Create an expression to represent the count() aggregate function
• count_distinct
  Create an expression to represent the count(distinct) aggregate function
• create_udaf
  Creates a new UDAF with a specific signature, state type and return type. The signature and state type must match the Accumulator's implementation.
• create_udf
  Convenience method to create a new user defined scalar function (UDF) with a specific signature and specific return type.
• create_udwf
  Creates a new UDWF with a specific signature, state type and return type.
• cube
  Create a grouping set for all combination of exprs
• current_date
  returns current UTC date as a DataType::Date32 value
• current_time
  returns current UTC time as a DataType::Time64 value
• date_bin
  coerces an arbitrary timestamp to the start of the nearest specified interval
• date_part
  extracts a subfield from the date
• date_trunc
  truncates the date to a specified level of precision
• decode
  decode theinput, using the encoding. encoding can be base64 or hex
• degrees
  converts radians to degrees
• digest
  compute the binary hash of input, using the algorithm
• encode
  encode the input, using the encoding. encoding can be base64 or hex
• exists
  Create an EXISTS subquery expression
• exp
  exponential
• factorial
  factorial
• find_in_set
  Returns a value in the range of 1 to N if the string str is in the string list strlist consisting of N substrings
• flatten
  flattens an array of arrays into a single array.
• floor
  nearest integer less than or equal to argument
• from_unixtime
  returns the unix time in format
• gcd
  greatest common divisor
• gen_range
  Returns a list of values in the range between start and stop with step.
• grouping_set
  Create a grouping set
• ident
  Create an unqualified column expression from the provided name, without normalizing the column.
• in_list
  Create an in_list expression
• in_subquery
  Create an IN subquery expression
• initcap
  converts the first letter of each word in string in uppercase and the remaining characters in lowercase
• is_false
  Create is false expression
• is_not_false
  Create is not false expression
• is_not_true
  Create is not true expression
• is_not_unknown
  Create is not unknown expression
• is_null
  Create is null expression
• is_true
  Create is true expression
• is_unknown
  Create is unknown expression
• isnan
  returns true if a given number is +NaN or -NaN otherwise returns false
• iszero
  returns true if a given number is +0.0 or -0.0 otherwise returns false
• lcm
  least common multiple
• left
  returns the first n characters in the string
• levenshtein
  Returns the Levenshtein distance between the two given strings
• lit
  Create a literal expression
• lit_timestamp_nano
  Create a literal timestamp expression
• ln
  natural logarithm
• log
  logarithm of a x for a particular base
• log2
  base 2 logarithm
• log10
  base 10 logarithm
• lower
  convert the string to lower case
• lpad
  fill up a string to the length by prepending the characters
• ltrim
  removes all characters, spaces by default, from the beginning of a string
• max
  Create an expression to represent the max() aggregate function
• md5
  returns the MD5 hash of a string
• median
  Calculate the median for expr.
• min
  Create an expression to represent the min() aggregate function
• nanvl
  returns x if x is not NaN otherwise returns y
• not
  Return a new expression with a logical NOT
• not_exists
  Create a NOT EXISTS subquery expression
• not_in_subquery
  Create a NOT IN subquery expression
• now
  returns current timestamp in nanoseconds, using the same value for all instances of now() in same statement
• nullif
  returns NULL if value1 equals value2; otherwise it returns value1. This can be used to perform the inverse operation of the COALESCE expression.
• octet_length
  returns the number of bytes of a string
• or
  Return a new expression with a logical OR
• out_ref_col
  Create an out reference column which hold a reference that has been resolved to a field outside of the current plan.
• overlay
  replace the substring of string that starts at the start’th character and extends for count characters with new substring
• pi
  Returns an approximate value of π
• placeholder
  Create placeholder value that will be filled in (such as $1)
• power
  base raised to the power of exponent
• radians
  converts degrees to radians
• random
  Returns a random value in the range 0.0 <= x < 1.0
• regexp_match
  matches a regular expression against a string and returns matched substrings.
• regexp_replace
  replace strings that match a regular expression
• repeat
  repeats the string to n times
• replace
  replaces all occurrences of from with to in the string
• reverse
  reverses the string
• right
  returns the last n characters in the string
• rollup
  Create a grouping set for rollup
• round
  round to nearest integer
• rpad
  fill up a string to the length by appending the characters
• rtrim
  removes all characters, spaces by default, from the end of a string
• scalar_subquery
  Create a scalar subquery expression
• sha224
  SHA-224 hash
• sha256
  SHA-256 hash
• sha384
  SHA-384 hash
• sha512
  SHA-512 hash
• signum
  sign of the argument (-1, 0, +1)
• sin
  sine
• sinh
  hyperbolic sine
• split_part
  splits a string based on a delimiter and picks out the desired field based on the index.
• sqrt
  square root of a number
• starts_with
  whether the string starts with the prefix
• stddev
  Create an expression to represent the stddev() aggregate function
• string_to_array
  splits a string based on a delimiter and returns an array of parts. Any parts matching the optional null_string will be replaced with NULL
• strpos
  finds the position from where the substring matches the string
• struct_fun
  returns a vector of fields from the struct
• substr
  substring from the position to the end
• substr_index
  Returns the substring from str before count occurrences of the delimiter
• substring
  substring from the position with length characters
• sum
  Create an expression to represent the sum() aggregate function
• tan
  tangent
• tanh
  hyperbolic tangent
• to_hex
  returns the hexdecimal representation of an integer
• to_timestamp
  converts a string and optional formats to a Timestamp(Nanoseconds, None)
• to_timestamp_micros
  converts a string and optional formats to a Timestamp(Microseconds, None)
• to_timestamp_millis
  converts a string and optional formats to a Timestamp(Milliseconds, None)
• to_timestamp_nanos
  converts a string and optional formats to a Timestamp(Nanoseconds, None)
• to_timestamp_seconds
  converts a string and optional formats to a Timestamp(Seconds, None)
• translate
  replaces the characters in from with the counterpart in to
• trim
  removes all characters, space by default from the string
• trunc
  truncate toward zero, with optional precision
• try_cast
  Create a try cast expression
• upper
  converts the string to upper case
• uuid
  returns uuid v4 as a string value
• when
  Create a CASE WHEN statement with boolean WHEN expressions and no base expression.
• wildcard
  Create an ‘*’ Expr::Wildcard expression that matches all columns