pub struct MemoryMappedCsvReader { /* private fields */ }Expand description
Memory-mapped CSV reader for efficient processing of large files.
Implementations§
Source§impl MemoryMappedCsvReader
impl MemoryMappedCsvReader
pub fn new(path: &Path) -> Result<Self>
pub fn new_with_tracker( path: &Path, memory_tracker: MemoryTracker, ) -> Result<Self>
Sourcepub fn read_chunk(
&self,
offset: u64,
chunk_size: usize,
) -> Result<(Vec<String>, usize)>
pub fn read_chunk( &self, offset: u64, chunk_size: usize, ) -> Result<(Vec<String>, usize)>
Read a chunk of the file starting at the given byte offset.
Returns (lines, actual_bytes_consumed) where actual_bytes_consumed
accounts for line-boundary trimming and may be less than chunk_size.
Sourcepub fn read_csv_chunk(
&self,
offset: u64,
chunk_size: usize,
has_headers: bool,
csv_config: Option<&CsvParserConfig>,
) -> Result<(Option<StringRecord>, Vec<StringRecord>, usize)>
pub fn read_csv_chunk( &self, offset: u64, chunk_size: usize, has_headers: bool, csv_config: Option<&CsvParserConfig>, ) -> Result<(Option<StringRecord>, Vec<StringRecord>, usize)>
Parse CSV records from memory-mapped data in chunks.
Returns (headers, records, actual_bytes_consumed).
Sourcepub fn ends_inside_quotes(
&self,
offset: u64,
csv_config: Option<&CsvParserConfig>,
) -> bool
pub fn ends_inside_quotes( &self, offset: u64, csv_config: Option<&CsvParserConfig>, ) -> bool
Whether the file ends inside a quoted field, scanning from offset on.
offset must be a record boundary, such as the offset of a chunk
Self::read_csv_chunk returned. Passing the last chunk’s offset keeps
the scan to that chunk. The CSV parser accepts a quote that is never
closed as a field running to the end of the file, so this is the only
sign of it.
Sourcepub fn estimate_row_count(&self) -> Result<usize>
pub fn estimate_row_count(&self) -> Result<usize>
Estimate the number of logical CSV records using the default parser configuration.
Retained as the backwards-compatible form of
Self::estimate_csv_record_count.
Sourcepub fn estimate_csv_record_count(
&self,
csv_config: Option<&CsvParserConfig>,
) -> Result<usize>
pub fn estimate_csv_record_count( &self, csv_config: Option<&CsvParserConfig>, ) -> Result<usize>
Estimate the number of logical CSV records in the file by sampling.
The sample uses Self::read_csv_chunk, so quoted multiline fields,
CR/LF/CRLF terminators, delimiters, and blank-record handling match the
records that incremental profiling will actually consume.
Sourcepub fn check_memory_leaks(&self) -> String
pub fn check_memory_leaks(&self) -> String
Check for memory leaks in the memory tracker.
Sourcepub fn get_memory_stats(&self) -> (usize, usize, usize)
pub fn get_memory_stats(&self) -> (usize, usize, usize)
Get memory usage statistics.