Struct StreamingIter

pub struct StreamingIter<'r, R> { /* private fields */ }

An iterator-like type to “stream” records from a reader.

This API returns records which use the StreamingBody type. This allows reading record headers and metadata without reading the bodies. Bodies can be read or skipped as desired.

This is streaming iterator is particularly useful for streams of records which are indefinite or contain and records of unknown size.

Implementations

impl<R: BufRead> StreamingIter<'_, R>

pub fn next_item( &mut self, ) -> Option<Result<Record<StreamingBody<'_, R>>, Error>>

Advance the stream to the next item.

Returns one of the following:

• Some(Ok(r))` is the next record read from the stream.
• Some(Err) indicates there was a read error.
• None indicates no more records are returned.

Examples found in repository

examples/read_filtered.rs (line 27)

fn main() -> std::io::Result<()> {
    let mut args = std::env::args_os().skip(1);

    let warc_name = args
        .next()
        .ok_or_else(|| usage_err!("compressed warc filename not supplied"))?;

    let filtered_file_names: Vec<_> = args.map(|s| s.to_string_lossy().to_string()).collect();
    if filtered_file_names.is_empty() {
        Err(usage_err!("one or more filtered file names not supplied"))?;
    }

    let mut file = WarcReader::from_path_gzip(warc_name)?;

    let mut count = 0;
    let mut skipped = 0;
    let mut stream_iter = file.stream_records();
    while let Some(record) = stream_iter.next_item() {
        let record = record.expect("read of headers ok");
        count += 1;
        match record.header(WarcHeader::TargetURI).map(|s| s.to_string()) {
            Some(v) if has_matching_filename(&v, &filtered_file_names) => {
                println!("Matches filename, skipping record");
                skipped += 1;
            }
            _ => {
                let buffered = record.into_buffered().expect("read of record ok");
                println!(
                    "Found record. Data:\n{}",
                    String::from_utf8_lossy(buffered.body())
                );
            }
        }
    }

    println!("Total records: {}\nSkipped records: {}", count, skipped);

    Ok(())
}