Module datafusion::common
source · Expand description
re-export of datafusion_common
crate
Modules§
- A complete, safe, native Rust implementation of Apache Arrow, a cross-language development platform for in-memory data.
- This module provides DataFusion specific casting functions that provide error handling. They are intended to “never fail” but provide an error message rather than a panic, as the corresponding kernels in arrow-rs such as
as_boolean_array
do. - Runtime configuration, via
ConfigOptions
- Types for plan display
- Options related to how files should be written
- Functionality used both on logical and physical plans
- WASM-compatible
Instant
wrapper. - Interval parsing logic
- Floating point rounding mode utility library TODO: Remove this custom implementation and the “libc” dependency when floating-point rounding mode manipulation functions become available in Rust.
- re-export of
datafusion_common_runtime
crate ScalarValue
: stores single values- This module provides data structures to represent statistics
- Utility functions to make testing DataFusion based crates easier
- This module provides common traits for visiting or rewriting tree data structures easily.
- This module provides the bisect function, which implements binary search.
Macros§
- Compares formatted output of a record batch with an expected vector of strings, with the result of pretty formatting record batches. This is a macro so errors appear on the correct line
- Compares formatted output of a record batch with an expected vector of strings in a way that order does not matter. This is a macro so errors appear on the correct line
- A macro to assert that one string is contained within another with a nice error message if they are not.
- A macro to assert that one string is NOT contained within another with a nice error message if they are are.
- Macro wraps
$ERR
to add backtrace feature - Macro wraps Err(
$ERR
) to add backtrace feature - A macro that wraps a configuration struct and automatically derives
Default
andConfigField
for it, allowing it to be used in theConfigOptions
configuration tree - Downcast an Arrow Array to a concrete type, return an
DataFusionError::Internal
if the cast is not possible. In normal usage of DataFusion the downcast should always succeed. - Macro wraps
$ERR
to add backtrace feature - Macro wraps Err(
$ERR
) to add backtrace feature - Convenience macro to create
ExtensionsOptions
. - This macro is used to control continuation behaviors during tree traversals based on the specified direction. Depending on
$DIRECTION
and the value of the given expression ($EXPR
), which should be a variant ofTreeNodeRecursion
, the macro results in the following behavior: - Macro wraps
$ERR
to add backtrace feature - Macro wraps Err(
$ERR
) to add backtrace feature - Macro wraps
$ERR
to add backtrace feature - Macro wraps Err(
$ERR
) to add backtrace feature - Macro wraps
$ERR
to add backtrace feature - Macro wraps Err(
$ERR
) to add backtrace feature - Macro wraps
$ERR
to add backtrace feature - Macro wraps Err(
$ERR
) to add backtrace feature - Unwrap an
Option
if possible. Otherwise return anDataFusionError::Internal
. In normal usage of DataFusion the unwrap should always succeed.
Structs§
- A named reference to a qualified field in a schema.
- Statistics for a column within a relation
- This object encapsulates a list of functional constraints:
- DFField wraps an Arrow field and adds an optional qualifier
- DFSchema wraps an Arrow schema and adds relation names.
- This object defines a functional dependence in the schema. A functional dependence defines a relationship between determinant keys and dependent columns. A determinant key is a column, or a set of columns, whose value uniquely determines values of some other (dependent) columns. If two rows have the same determinant key, dependent columns in these rows are necessarily the same. If the determinant key is unique, the set of dependent columns is equal to the entire schema and the determinant key can serve as a primary key. Note that a primary key may “downgrade” into a determinant key due to an operation such as a join, and this object is used to track dependence relationships in such cases. For more information on functional dependencies, see: https://www.scaler.com/topics/dbms/functional-dependency-in-dbms/
- This object encapsulates all functional dependencies in a given relation.
- A resolved path to a table of the form “catalog.schema.table”
- Statistics for a relation Fields are optional and can be inexact because the sources sometimes provide approximate estimates for performance reasons and the transformations output are not always predictable.
- Options for unnesting a column that contains a list type, replicating values in the other, non nested rows.
Enums§
- This object defines a constraint on a table.
- DataFusion error
- Describes functional dependency mode.
- Readable file type
- Join constraint
- Join side. Stores the referred table side during calculations
- Join type
- The parameter value corresponding to the placeholder
- A dynamically typed, nullable single value.
- Schema-related errors
TableReference
s represent a multi part identifier (path) to a table that may require further resolution.
Constants§
- The default file extension of arrow files
- The default file extension of avro files
- The default file extension of csv files
- The default file extension of json files
- The default file extension of parquet files
Traits§
- Provides schema information needed by certain methods of
Expr
(defined in the datafusion-common crate). - Define each
FileType
/FileCompressionType
’s extension - Trait used to map a NativeType to a ScalarValue
- DataFusion-specific extensions to
Schema
. - Convenience trait to convert Schema like things to DFSchema and DFSchemaRef with fewer keystrokes
Functions§
- Calculates functional dependencies for aggregate output, when there is a GROUP BY expression.
- Create a “field not found” DataFusion::SchemaError
- Returns indices for the minimal subset of GROUP BY expressions that are functionally equivalent to the original set of GROUP BY expressions.
- Returns target indices, for the determinant keys that are inside group by expressions.
- Applies an optional projection to a
SchemaRef
, returning the projected schema - Convenience wrapper over
field_not_found
for when there is no qualifier
Type Aliases§
- A reference-counted reference to a DFSchema.
- This is a
TableReference
that has ’static lifetime (aka it owns the underlying string) - Result type for operations that could result in an DataFusionError
- Result type for operations that could result in an DataFusionError and needs to be shared (wrapped into
Arc
).