Struct deltalake::datafusion::datasource::physical_plan::CsvOpener

pub struct CsvOpener { /* private fields */ }

A FileOpener that opens a CSV file and yields a FileOpenFuture

Implementations

impl CsvOpener

pub fn new( config: Arc<CsvConfig>, file_compression_type: FileCompressionType ) -> CsvOpener

Returns a CsvOpener

Trait Implementations

impl FileOpener for CsvOpener

fn open( &self, file_meta: FileMeta ) -> Result<Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<RecordBatch, ArrowError>> + Send>>, DataFusionError>> + Send>>, DataFusionError>

Open a partitioned CSV file.

If file_meta.range is None, the entire file is opened. If file_meta.range is Some(FileRange {start, end}), this signifies that the partition corresponds to the byte range [start, end) within the file.

Note: start or end might be in the middle of some lines. In such cases, the following rules are applied to determine which lines to read:

1. The first line of the partition is the line in which the index of the first character >= start.
2. The last line of the partition is the line in which the byte at position end - 1 resides.

Examples: Consider the following partitions enclosed by braces {}:

{A,1,2,3,4,5,6,7,8,9\n A,1,2,3,4,5,6,7,8,9\n} A,1,2,3,4,5,6,7,8,9\n The lines read would be: [0, 1]

A,{1,2,3,4,5,6,7,8,9\n A,1,2,3,4,5,6,7,8,9\n A},1,2,3,4,5,6,7,8,9\n The lines read would be: [1, 2]