Struct Record 

pub struct Record<T: BodyKind> { /* private fields */ }

A single WARC record.

A record can be constructed by a RecordBuilder, or by reading from a stream.

The associated type T indicates the representation of this record’s body.

A record is guaranteed to be valid according to the specification it conforms to, except:

• The validity of the WARC-Record-ID header is not checked
• Date information not in the UTC timezone will be silently converted to UTC

Use the Display trait to generate the formatted representation.

Implementations

impl<T: BodyKind> Record<T>

pub fn new() -> Record<EmptyBody>

Create a new empty record with default values.

Using a RecordBuilder is more efficient when creating records from known data.

The record returned contains an empty body, and the following fields:

• WARC-Record-ID: generated by generate_record_id()
• WARC-Date: the current moment in time
• WARC-Type: resource
• WARC-Content-Length: 0

Examples found in repository

examples/write_file.rs (line 10)

fn main() -> Result<(), std::io::Error> {
    let date = Utc::now();
    let body = format!("wrote to the file on {}", date);
    let body = body.into_bytes();

    let mut headers = Record::<BufferedBody>::new();
    headers.set_warc_type(RecordType::WarcInfo);
    headers.set_date(date);
    headers
        .set_header(WarcHeader::IPAddress, "127.0.0.1")
        .expect("BUG: should be a valid IP address");
    let record = headers.add_body(body);

    let mut file = WarcWriter::from_path("warc_example.warc")?;

    let bytes_written = file.write(&record)?;

    println!("{} bytes written.", bytes_written);

    Ok(())
}

pub fn with_body<B: Into<Vec<u8>>>(body: B) -> Record<BufferedBody>

Create a new empty record with a known body.

Using a RecordBuilder is more efficient when creating records from known data.

The record returned contains the passed body buffer, and the following fields:

• WARC-Record-ID: generated by generate_record_id()
• WARC-Date: the current moment in time
• WARC-Type: resource
• WARC-Content-Length: body.len()

pub fn generate_record_id() -> String

Generate and return a new value suitable for use in the WARC-Record-ID header.

Compatibility

The standard only places a small number of constraints on this field:

1. This value is globally unique “for its period of use”
2. This value is a valid URI
3. This value “clearly indicate[s] a documented and registered scheme to which it conforms.”

These guarantees will be upheld by all generated outputs, where the “period of use” is presumed to be indefinite and unlimited.

However, any specific algorithm used to generate values is not part of the crate’s public API for purposes of semantic versioning.

Implementation

The current implementation generates random values based on UUID version 4.

Examples found in repository

examples/hello_warc.rs (line 14)

fn main() {
    let body = "hello warc! 👋".to_owned();

    let headers = RawRecordHeader {
        version: "1.0".to_owned(),
        headers: vec![
            (
                WarcHeader::RecordID,
                Record::<BufferedBody>::generate_record_id().into_bytes(),
            ),
            (
                WarcHeader::WarcType,
                RecordType::WarcInfo.to_string().into_bytes(),
            ),
            (
                WarcHeader::Date,
                Utc::now()
                    .to_rfc3339_opts(SecondsFormat::Secs, true)
                    .into_bytes(),
            ),
            (WarcHeader::IPAddress, "127.0.0.1".to_owned().into_bytes()),
            (
                WarcHeader::ContentLength,
                body.len().to_string().into_bytes(),
            ),
        ]
        .into_iter()
        .collect(),
    };

    println!("{}{}", headers, body);
}

examples/write_raw.rs (line 16)

fn main() -> Result<(), std::io::Error> {
    let date = Utc::now().to_rfc3339_opts(SecondsFormat::Secs, true);
    let body = format!("wrote to the file on {}", date);
    let body = body.into_bytes();

    let headers = RawRecordHeader {
        version: "1.0".to_owned(),
        headers: vec![
            (
                WarcHeader::RecordID,
                Record::<BufferedBody>::generate_record_id().into_bytes(),
            ),
            (
                WarcHeader::WarcType,
                RecordType::WarcInfo.to_string().into_bytes(),
            ),
            (WarcHeader::Date, date.into_bytes()),
            (WarcHeader::IPAddress, "127.0.0.1".to_owned().into_bytes()),
            (
                WarcHeader::ContentLength,
                body.len().to_string().into_bytes(),
            ),
        ]
        .into_iter()
        .collect(),
    };

    let mut file = WarcWriter::from_path("warc_example.warc")?;

    let bytes_written = file.write_raw(headers, &body)?;

    println!("{} bytes written.", bytes_written);

    Ok(())
}

examples/write_gzip.rs (line 16)

fn main() -> Result<(), std::io::Error> {
    let date = Utc::now().to_rfc3339_opts(SecondsFormat::Secs, true);
    let body = format!("wrote to the file on {}", date);
    let body = body.into_bytes();

    let headers = RawRecordHeader {
        version: "1.0".to_owned(),
        headers: vec![
            (
                WarcHeader::RecordID,
                Record::<BufferedBody>::generate_record_id().into_bytes(),
            ),
            (
                WarcHeader::WarcType,
                RecordType::WarcInfo.to_string().into_bytes(),
            ),
            (WarcHeader::Date, date.into_bytes()),
            (WarcHeader::IPAddress, "127.0.0.1".to_owned().into_bytes()),
            (
                WarcHeader::ContentLength,
                body.len().to_string().into_bytes(),
            ),
        ]
        .into_iter()
        .collect(),
    };

    let mut file = WarcWriter::from_path_gzip("warc_example.warc.gz")?;

    let bytes_written = file.write_raw(headers, &body)?;

    // NB: the compression stream must be finish()ed, or the file will be truncated
    let gzip_stream = file.into_inner()?;
    gzip_stream.finish().into_result()?;

    println!("{} bytes written.", bytes_written);

    Ok(())
}

pub fn warc_version(&self) -> &str

Return the WARC version string of this record.

pub fn set_warc_version<S: Into<String>>(&mut self, id: S)

Set the WARC version string of this record.

pub fn warc_id(&self) -> &str

Return the WARC-Record-ID header for this record.

Examples found in repository

examples/read_file.rs (line 13)

fn main() -> Result<(), std::io::Error> {
    let file = WarcReader::from_path("warc_example.warc")?;

    let mut count = 0;
    for record in file.iter_records() {
        count += 1;
        match record {
            Err(err) => println!("ERROR: {}\r\n", err),
            Ok(record) => {
                println!("{}: {}", WarcHeader::RecordID, record.warc_id(),);
                println!("{}: {}", WarcHeader::Date, record.date(),);
                println!();
            }
        }
    }

    println!("Total records: {}", count);

    Ok(())
}

examples/read_gzip.rs (line 13)

fn main() -> Result<(), std::io::Error> {
    let file = WarcReader::from_path_gzip("warc_example.warc.gz")?;

    let mut count = 0;
    for record in file.iter_records() {
        count += 1;
        match record {
            Err(err) => println!("ERROR: {}\r\n", err),
            Ok(record) => {
                println!("{}: {}", WarcHeader::RecordID, record.warc_id());
                println!("{}: {}", WarcHeader::Date, record.date());
                println!();
            }
        }
    }

    println!("Total records: {}", count);

    Ok(())
}

pub fn set_warc_id<S: Into<String>>(&mut self, id: S)

Set the WARC-Record-ID header for this record.

Note that this value is not checked for validity.

pub fn warc_type(&self) -> &RecordType

Return the WARC-Type header for this record.

pub fn set_warc_type(&mut self, type_: RecordType)

Set the WARC-Type header for this record.

Examples found in repository

examples/write_file.rs (line 11)

fn main() -> Result<(), std::io::Error> {
    let date = Utc::now();
    let body = format!("wrote to the file on {}", date);
    let body = body.into_bytes();

    let mut headers = Record::<BufferedBody>::new();
    headers.set_warc_type(RecordType::WarcInfo);
    headers.set_date(date);
    headers
        .set_header(WarcHeader::IPAddress, "127.0.0.1")
        .expect("BUG: should be a valid IP address");
    let record = headers.add_body(body);

    let mut file = WarcWriter::from_path("warc_example.warc")?;

    let bytes_written = file.write(&record)?;

    println!("{} bytes written.", bytes_written);

    Ok(())
}

pub fn date(&self) -> &DateTime<Utc>

Return the WARC-Date header for this record.

Examples found in repository

examples/read_file.rs (line 14)

fn main() -> Result<(), std::io::Error> {
    let file = WarcReader::from_path("warc_example.warc")?;

    let mut count = 0;
    for record in file.iter_records() {
        count += 1;
        match record {
            Err(err) => println!("ERROR: {}\r\n", err),
            Ok(record) => {
                println!("{}: {}", WarcHeader::RecordID, record.warc_id(),);
                println!("{}: {}", WarcHeader::Date, record.date(),);
                println!();
            }
        }
    }

    println!("Total records: {}", count);

    Ok(())
}

examples/read_gzip.rs (line 14)

fn main() -> Result<(), std::io::Error> {
    let file = WarcReader::from_path_gzip("warc_example.warc.gz")?;

    let mut count = 0;
    for record in file.iter_records() {
        count += 1;
        match record {
            Err(err) => println!("ERROR: {}\r\n", err),
            Ok(record) => {
                println!("{}: {}", WarcHeader::RecordID, record.warc_id());
                println!("{}: {}", WarcHeader::Date, record.date());
                println!();
            }
        }
    }

    println!("Total records: {}", count);

    Ok(())
}

pub fn set_date(&mut self, date: DateTime<Utc>)

Set the WARC-Date header for this record.

Examples found in repository

examples/write_file.rs (line 12)

fn main() -> Result<(), std::io::Error> {
    let date = Utc::now();
    let body = format!("wrote to the file on {}", date);
    let body = body.into_bytes();

    let mut headers = Record::<BufferedBody>::new();
    headers.set_warc_type(RecordType::WarcInfo);
    headers.set_date(date);
    headers
        .set_header(WarcHeader::IPAddress, "127.0.0.1")
        .expect("BUG: should be a valid IP address");
    let record = headers.add_body(body);

    let mut file = WarcWriter::from_path("warc_example.warc")?;

    let bytes_written = file.write(&record)?;

    println!("{} bytes written.", bytes_written);

    Ok(())
}

pub fn truncated_type(&self) -> &Option<TruncatedType>

Return the WARC-Truncated header for this record.

pub fn set_truncated_type(&mut self, truncated_type: TruncatedType)

Set the WARC-Truncated header for this record.

pub fn clear_truncated_type(&mut self)

Remove the WARC-Truncated header for this record.

pub fn header(&self, header: WarcHeader) -> Option<Cow<'_, str>>

Return the WARC header requested if present in this record, or None.

Examples found in repository

examples/read_filtered.rs (line 30)

fn main() -> std::io::Result<()> {
    let mut args = std::env::args_os().skip(1);

    let warc_name = args
        .next()
        .ok_or_else(|| usage_err!("compressed warc filename not supplied"))?;

    let filtered_file_names: Vec<_> = args.map(|s| s.to_string_lossy().to_string()).collect();
    if filtered_file_names.is_empty() {
        Err(usage_err!("one or more filtered file names not supplied"))?;
    }

    let mut file = WarcReader::from_path_gzip(warc_name)?;

    let mut count = 0;
    let mut skipped = 0;
    let mut stream_iter = file.stream_records();
    while let Some(record) = stream_iter.next_item() {
        let record = record.expect("read of headers ok");
        count += 1;
        match record.header(WarcHeader::TargetURI).map(|s| s.to_string()) {
            Some(v) if has_matching_filename(&v, &filtered_file_names) => {
                println!("Matches filename, skipping record");
                skipped += 1;
            }
            _ => {
                let buffered = record.into_buffered().expect("read of record ok");
                println!(
                    "Found record. Data:\n{}",
                    String::from_utf8_lossy(buffered.body())
                );
            }
        }
    }

    println!("Total records: {}\nSkipped records: {}", count, skipped);

    Ok(())
}

pub fn set_header<V>( &mut self, header: WarcHeader, value: V, ) -> Result<Option<Cow<'_, str>>, WarcError>

where V: Into<String>,

Set a WARC header in this record, returning the previous value if present.

Errors

If setting a header whose value has a well-formedness test, an error is returned if the value is not well-formed.

Examples found in repository

examples/write_file.rs (line 14)

fn main() -> Result<(), std::io::Error> {
    let date = Utc::now();
    let body = format!("wrote to the file on {}", date);
    let body = body.into_bytes();

    let mut headers = Record::<BufferedBody>::new();
    headers.set_warc_type(RecordType::WarcInfo);
    headers.set_date(date);
    headers
        .set_header(WarcHeader::IPAddress, "127.0.0.1")
        .expect("BUG: should be a valid IP address");
    let record = headers.add_body(body);

    let mut file = WarcWriter::from_path("warc_example.warc")?;

    let bytes_written = file.write(&record)?;

    println!("{} bytes written.", bytes_written);

    Ok(())
}

pub fn content_length(&self) -> u64

Return the Content-Length header for this record.

This value is guaranteed to match the actual length of the body.

impl Record<EmptyBody>

pub fn add_body<B: Into<Vec<u8>>>(self, body: B) -> Record<BufferedBody>

Add a known body to this record, transforming it into a buffered body record.

Examples found in repository

examples/write_file.rs (line 16)

fn main() -> Result<(), std::io::Error> {
    let date = Utc::now();
    let body = format!("wrote to the file on {}", date);
    let body = body.into_bytes();

    let mut headers = Record::<BufferedBody>::new();
    headers.set_warc_type(RecordType::WarcInfo);
    headers.set_date(date);
    headers
        .set_header(WarcHeader::IPAddress, "127.0.0.1")
        .expect("BUG: should be a valid IP address");
    let record = headers.add_body(body);

    let mut file = WarcWriter::from_path("warc_example.warc")?;

    let bytes_written = file.write(&record)?;

    println!("{} bytes written.", bytes_written);

    Ok(())
}

pub fn add_fixed_stream<'r, R: Read + 'r>( self, stream: &'r mut R, len: &'r mut u64, ) -> Result<Record<StreamingBody<'r, R>>>

Add a streaming body to this record, whose expected size may not match the actual stream length.

impl Record<BufferedBody>

pub fn strip_body(self) -> Record<EmptyBody>

Strip the body from this record.

pub fn body(&self) -> &[u8]

Return the body of this record.

Examples found in repository

examples/read_filtered.rs (line 39)

fn main() -> std::io::Result<()> {
    let mut args = std::env::args_os().skip(1);

    let warc_name = args
        .next()
        .ok_or_else(|| usage_err!("compressed warc filename not supplied"))?;

    let filtered_file_names: Vec<_> = args.map(|s| s.to_string_lossy().to_string()).collect();
    if filtered_file_names.is_empty() {
        Err(usage_err!("one or more filtered file names not supplied"))?;
    }

    let mut file = WarcReader::from_path_gzip(warc_name)?;

    let mut count = 0;
    let mut skipped = 0;
    let mut stream_iter = file.stream_records();
    while let Some(record) = stream_iter.next_item() {
        let record = record.expect("read of headers ok");
        count += 1;
        match record.header(WarcHeader::TargetURI).map(|s| s.to_string()) {
            Some(v) if has_matching_filename(&v, &filtered_file_names) => {
                println!("Matches filename, skipping record");
                skipped += 1;
            }
            _ => {
                let buffered = record.into_buffered().expect("read of record ok");
                println!(
                    "Found record. Data:\n{}",
                    String::from_utf8_lossy(buffered.body())
                );
            }
        }
    }

    println!("Total records: {}\nSkipped records: {}", count, skipped);

    Ok(())
}

pub fn body_mut(&mut self) -> &mut [u8]

Return a reference to mutate the body of this record, but without changing its length.

To update the body of the record or change its length, use the replace_body method instead.

pub fn replace_body<V: Into<Vec<u8>>>(&mut self, new_body: V)

Replace the body of this record with the given body.

pub fn into_raw_parts(self) -> (RawRecordHeader, Vec<u8>)

Transform this record into a raw record containing the same data.

impl<'t, T: Read + 't> Record<StreamingBody<'t, T>>

pub fn into_buffered(self) -> Result<Record<BufferedBody>>

Returns a record with a buffered body by collecting the streaming body.

Errors

This method can fail if the underlying stream returns an error. If this happens, the state of the stream is not guaranteed.

Examples found in repository

examples/read_filtered.rs (line 36)

fn main() -> std::io::Result<()> {
    let mut args = std::env::args_os().skip(1);

    let warc_name = args
        .next()
        .ok_or_else(|| usage_err!("compressed warc filename not supplied"))?;

    let filtered_file_names: Vec<_> = args.map(|s| s.to_string_lossy().to_string()).collect();
    if filtered_file_names.is_empty() {
        Err(usage_err!("one or more filtered file names not supplied"))?;
    }

    let mut file = WarcReader::from_path_gzip(warc_name)?;

    let mut count = 0;
    let mut skipped = 0;
    let mut stream_iter = file.stream_records();
    while let Some(record) = stream_iter.next_item() {
        let record = record.expect("read of headers ok");
        count += 1;
        match record.header(WarcHeader::TargetURI).map(|s| s.to_string()) {
            Some(v) if has_matching_filename(&v, &filtered_file_names) => {
                println!("Matches filename, skipping record");
                skipped += 1;
            }
            _ => {
                let buffered = record.into_buffered().expect("read of record ok");
                println!(
                    "Found record. Data:\n{}",
                    String::from_utf8_lossy(buffered.body())
                );
            }
        }
    }

    println!("Total records: {}\nSkipped records: {}", count, skipped);

    Ok(())
}

Trait Implementations

impl Clone for Record<BufferedBody>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Clone for Record<EmptyBody>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Debug + BodyKind> Debug for Record<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Record<BufferedBody>

fn default() -> Record<BufferedBody>

Returns the “default value” for a type.

impl Default for Record<EmptyBody>

fn default() -> Record<EmptyBody>

Returns the “default value” for a type.

impl Display for Record<BufferedBody>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Record<EmptyBody>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: PartialEq + BodyKind> PartialEq for Record<T>

fn eq(&self, other: &Record<T>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<'t, T: Read + 't> Read for Record<StreamingBody<'t, T>>

fn read(&mut self, dst: &mut [u8]) -> Result<usize, Error>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Reads all bytes until EOF in this source, placing them into buf.

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Reads all bytes until EOF in this source, appending them to buf.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Reads the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact(&mut self, cursor: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Reads the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

where Self: Sized,

Creates a “by reference” adaptor for this instance of Read.

fn bytes(self) -> Bytes<Self>

where Self: Sized,

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>

where R: Read, Self: Sized,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

where Self: Sized,

Creates an adapter which will read at most limit bytes from it.

impl TryFrom<RawRecordHeader> for Record<EmptyBody>

type Error = Error

The type returned in the event of a conversion error.

fn try_from(headers: RawRecordHeader) -> Result<Self, WarcError>

Performs the conversion.

impl<T: BodyKind> StructuralPartialEq for Record<T>