Module file_stream

A generic stream over file format readers that can be used by any file format that read its files from start to end.

Note: Most traits here need to be marked Sync + Send to be compliant with the SendableRecordBatchStream trait.

Structs

FileStream

A stream that iterates record batch by record batch, file over file.

FileStreamMetrics

Metrics for FileStream

StartableTime

A timer that can be started and stopped.

Enums

FileStreamState

NextOpen

Represents the state of the next FileOpenFuture. Since we need to poll this future while scanning the current file, we need to store the result if it is ready

OnError

Describes the behavior of the FileStream if file opening or scanning fails

Traits

FileOpener

Generic API for opening a file using an ObjectStore and resolving to a stream of RecordBatch

Type Aliases

FileOpenFuture

A fallible future that resolves to a stream of RecordBatch