Struct FileStreamMetrics

pub struct FileStreamMetrics {
    pub time_opening: StartableTime,
    pub time_scanning_until_data: StartableTime,
    pub time_scanning_total: StartableTime,
    pub time_processing: StartableTime,
    pub file_open_errors: Count,
    pub file_scan_errors: Count,
}

Metrics for FileStream

Note that all of these metrics are in terms of wall clock time (not cpu time) so they include time spent waiting on I/O as well as other operators.

Fields

time_opening: StartableTime

Wall clock time elapsed for file opening.

Time between when FileOpener::open is called and when the FileStream receives a stream for reading.

If there are multiple files being scanned, the stream will open the next file in the background while scanning the current file. This metric will only capture time spent opening while not also scanning. FileStream: https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/physical_plan/file_stream.rs

time_scanning_until_data: StartableTime

Wall clock time elapsed for file scanning + first record batch of decompression + decoding

Time between when the FileStream requests data from the stream and when the first RecordBatch is produced. FileStream: https://github.com/apache/datafusion/blob/main/datafusion/core/src/datasource/physical_plan/file_stream.rs

time_scanning_total: StartableTime

Total elapsed wall clock time for scanning + record batch decompression / decoding

Sum of time between when the FileStream requests data from the stream and when a RecordBatch is produced for all record batches in the stream. Note that this metric also includes the time of the parent operator’s execution.

time_processing: StartableTime

Wall clock time elapsed for data decompression + decoding

Time spent waiting for the FileStream’s input.

file_open_errors: Count

Count of errors opening file.

If using OnError::Skip this will provide a count of the number of files which were skipped and will not be included in the scan results.

file_scan_errors: Count

Count of errors scanning file

If using OnError::Skip this will provide a count of the number of files which were skipped and will not be included in the scan results.

Implementations

impl FileStreamMetrics

pub fn new(metrics: &ExecutionPlanMetricsSet, partition: usize) -> Self