Struct FileStreamMetrics 

pub struct FileStreamMetrics {
    pub time_opening: StartableTime,
    pub time_scanning_until_data: StartableTime,
    pub time_scanning_total: StartableTime,
    pub time_processing: Time,
    pub file_open_errors: Count,
    pub file_scan_errors: Count,
    pub files_opened: Count,
    pub files_processed: Count,
}

Metrics for FileStream

Note that all of these metrics are in terms of wall clock time (not cpu time) so they include time spent waiting on I/O as well as other operators.

Fields

time_opening: StartableTime

Wall clock time elapsed for file opening.

Time between when FileOpener::open is called and when the FileStream receives a stream for reading.

time_scanning_until_data: StartableTime

Wall clock time elapsed for file scanning + first record batch of decompression + decoding

Time between when the FileStream requests data from the stream and when the first RecordBatch is produced.

time_scanning_total: StartableTime

Total elapsed wall clock time for scanning + record batch decompression / decoding

Sum of time between when the FileStream requests data from the stream and when a RecordBatch is produced for all record batches in the stream. Note that this metric also includes the time of the parent operator’s execution.

time_processing: Time

Wall clock time elapsed for data decompression + decoding

Time spent waiting for the FileStream’s input.

file_open_errors: Count

Count of errors opening file.

If using OnError::Skip this will provide a count of the number of files which were skipped and will not be included in the scan results.

file_scan_errors: Count

Count of errors scanning file

If using OnError::Skip this will provide a count of the number of files which were skipped and will not be included in the scan results.

files_opened: Count

Count of files successfully opened or evaluated for processing. At t=end (completion of a query) this is equal to files_opened, and both values are equal to the total number of files in the query; unless the query itself fails. This value will always be greater than or equal to files_open. Note that this value does not mean the file was actually scanned. We increment this value for any processing of a file, even if that processing is discarding it because we hit a LIMIT (in this case files_opened and files_processed are both incremented at the same time).

files_processed: Count

Count of files completely processed / closed (opened, pruned, or skipped due to limit). At t=0 (the beginning of a query) this is 0. At t=end (completion of a query) this is equal to files_opened, and both values are equal to the total number of files in the query; unless the query itself fails. This value will always be less than or equal to files_open. We increment this value for any processing of a file, even if that processing is discarding it because we hit a LIMIT (in this case files_opened and files_processed are both incremented at the same time).

Implementations

impl FileStreamMetrics

pub fn new(metrics: &ExecutionPlanMetricsSet, partition: usize) -> Self