Struct CsvExec

pub struct CsvExec { /* private fields */ }

👎Deprecated since 46.0.0: use DataSourceExec instead

Old Csv source, deprecated with DataSourceExec implementation and CsvSource

See examples on CsvSource

Implementations

impl CsvExec

pub fn new( base_config: FileScanConfig, has_header: bool, delimiter: u8, quote: u8, terminator: Option<u8>, escape: Option<u8>, comment: Option<u8>, newlines_in_values: bool, file_compression_type: FileCompressionType, ) -> Self

Create a new CSV reader execution plan provided base and specific configurations

pub fn builder(file_scan_config: FileScanConfig) -> CsvExecBuilder

Return a CsvExecBuilder.

See example on CsvExec and CsvExecBuilder for specifying CSV table options.

pub fn base_config(&self) -> &FileScanConfig

Ref to the base configs

pub fn has_header(&self) -> bool

true if the first line of each file is a header

pub fn newlines_in_values(&self) -> bool

Specifies whether newlines in (quoted) values are supported.

Parsing newlines in quoted values may be affected by execution behaviour such as parallel file scanning. Setting this to true ensures that newlines in values are parsed successfully, which may reduce performance.

The default behaviour depends on the datafusion.catalog.newlines_in_values setting.

Trait Implementations

impl Clone for CsvExec

fn clone(&self) -> CsvExec

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CsvExec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DisplayAs for CsvExec

fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format according to DisplayFormatType, used when verbose representation looks different from the default one

impl ExecutionPlan for CsvExec

fn as_any(&self) -> &dyn Any

Return a reference to Any that can be used for downcasting

fn repartitioned( &self, target_partitions: usize, config: &ConfigOptions, ) -> Result<Option<Arc<dyn ExecutionPlan>>>

Redistribute files across partitions according to their size See comments on FileGroupPartitioner for more detail.

Return None if can’t get repartitioned (empty, compressed file, or newlines_in_values set).

fn name(&self) -> &'static str

Short name for the ExecutionPlan, such as ‘DataSourceExec’.

fn properties(&self) -> &PlanProperties

Return properties of the output of the ExecutionPlan, such as output ordering(s), partitioning information etc.

fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>>

Get a list of children ExecutionPlans that act as inputs to this plan. The returned list will be empty for leaf nodes such as scans, will contain a single value for unary nodes, or two values for binary nodes (such as joins).

fn with_new_children( self: Arc<Self>, _: Vec<Arc<dyn ExecutionPlan>>, ) -> Result<Arc<dyn ExecutionPlan>>

Returns a new ExecutionPlan where all existing children were replaced by the children, in order

fn execute( &self, partition: usize, context: Arc<TaskContext>, ) -> Result<SendableRecordBatchStream>

Begin execution of partition, returning a Stream of RecordBatches.

fn statistics(&self) -> Result<Statistics>

Returns statistics for this ExecutionPlan node. If statistics are not available, should return Statistics::new_unknown (the default), not an error.

fn metrics(&self) -> Option<MetricsSet>

Return a snapshot of the set of Metrics for this ExecutionPlan. If no Metrics are available, return None.

fn fetch(&self) -> Option<usize>

Gets the fetch count for the operator, None means there is no fetch.

fn with_fetch(&self, limit: Option<usize>) -> Option<Arc<dyn ExecutionPlan>>

Returns a fetching variant of this ExecutionPlan node, if it supports fetch limits. Returns None otherwise.

fn try_swapping_with_projection( &self, projection: &ProjectionExec, ) -> Result<Option<Arc<dyn ExecutionPlan>>>

Attempts to push down the given projection into the input of this ExecutionPlan.

fn static_name() -> &'static str

where Self: Sized,

Short name for the ExecutionPlan, such as ‘DataSourceExec’. Like name but can be called without an instance.

fn schema(&self) -> Arc<Schema>

Get the schema for this execution plan

fn check_invariants( &self, _check: InvariantLevel, ) -> Result<(), DataFusionError>

Returns an error if this individual node does not conform to its invariants. These invariants are typically only checked in debug mode.

fn required_input_distribution(&self) -> Vec<Distribution>

Specifies the data distribution requirements for all the children for this ExecutionPlan, By default it’s [Distribution::UnspecifiedDistribution] for each child,

fn required_input_ordering(&self) -> Vec<Option<LexRequirement>>

Specifies the ordering required for all of the children of this ExecutionPlan.

fn maintains_input_order(&self) -> Vec<bool>

Returns false if this ExecutionPlan’s implementation may reorder rows within or between partitions.

fn benefits_from_input_partitioning(&self) -> Vec<bool>

Specifies whether the ExecutionPlan benefits from increased parallelization at its input for each child.

fn supports_limit_pushdown(&self) -> bool

Returns true if a limit can be safely pushed down through this ExecutionPlan node.

fn cardinality_effect(&self) -> CardinalityEffect

Gets the effect on cardinality, if known