bytesreader

bytesreader is a growable buffered reader for records, including those larger than its input buffer. Keep data in the buffer while reading more, then borrow a slice or take ownership of it as a BytesMut.

The crate works on stable Rust. The optional nightly feature uses Read::read_buf to fill the buffer and requires a nightly toolchain.

Performance

For owned records smaller than roughly 1 KiB, BufReader with to_vec() was often faster in these tests. For larger records, BytesReader::take_bytes can be faster: when you consume the selected bytes, it can return them as a BytesMut without copying.

The benchmarks below read newline-delimited records from cached files. For each selected payload size, the tables show the range of median times across three runs. An asterisk marks the lowest median of those three run medians; close results may be measurement noise. Borrowed and owned results are compared separately. The BufReader paths use fill_buf, slice the record, and call consume.

When a record spans reads, the BufReader examples collect its parts in a temporary Vec. One copies the completed record and reuses the temporary buffer; the other returns it with mem::take. When a record fits in one read, both copy it into a Vec.

Borrowing a slice:

Owned records, grouped by payload so you can compare take_bytes and to_vec() directly:

The tests used Rust 1.101.0-nightly on Linux under WSL2, with an Intel i7-13700KF. Initial buffers were 8 KiB for payloads through 1500 B and 1 MiB for larger payloads. Each record included a newline, and owned results were dropped before reading the next record. Keeping results longer may change performance. Each size was measured in three runs of 13 rounds, with four passes per round. To run the per-size comparison on your machine:

cargo +nightly bench --bench reading --features nightly -- --summary=no

See benchmarks for the other options.

Reading example

Use take_view(range, consume) to borrow part of the buffer, or take_bytes to take it as an owned BytesMut. Both let you choose how much input to consume. To copy a view into a Vec<u8>, call to_vec() on it.

For example, given <abc>tail, take_view(1..4, Some(5)) returns abc and leaves tail. Pass None instead of Some(n) to consume through the end of the selected range.

BytesReader also provides the stable methods of std::io::BufReader, including Read and BufRead.

The LineReader below reads lines, calling read_more when a line spans reads. Its read_view method borrows a line, read_owned copies it into a Vec<u8>, and read_bytes returns a BytesMut that can outlive the reader. The newline is consumed but not returned, and the last line does not need one.

use std::io::{self, Read};

use bytes::BytesMut;
use bytesreader::BytesReader;

struct LineReader<R> {
    reader: BytesReader<R>,
}

impl<R: Read> LineReader<R> {
    fn with_capacity(capacity: usize, source: R) -> Self {
        Self { reader: BytesReader::with_capacity(capacity, source) }
    }

    fn line_bounds(&mut self) -> io::Result<Option<(usize, usize)>> {
        let mut scanned = 0;
        loop {
            if let Some(offset) = self.reader.buffer()[scanned..].iter().position(|&b| b == b'\n') {
                let end = scanned + offset;
                return Ok(Some((end, end + 1)));
            }
            scanned = self.reader.buffer().len();
            match self.reader.read_more() {
                Ok(0) => return Ok(self.reader.has_remaining().then_some((scanned, scanned))),
                Ok(_) => {}
                Err(error) if error.kind() == io::ErrorKind::Interrupted => continue,
                Err(error) => return Err(error),
            }
        }
    }

    fn read_bytes(&mut self) -> io::Result<Option<BytesMut>> {
        let Some((end, consume)) = self.line_bounds()? else {
            return Ok(None);
        };
        Ok(Some(self.reader.take_bytes(..end, Some(consume))))
    }

    fn read_view(&mut self) -> io::Result<Option<&[u8]>> {
        let Some((end, consume)) = self.line_bounds()? else {
            return Ok(None);
        };
        Ok(Some(self.reader.take_view(..end, Some(consume))))
    }

    fn read_owned(&mut self) -> io::Result<Option<Vec<u8>>> {
        Ok(self.read_view()?.map(|record| record.to_vec()))
    }
}

let mut reader = LineReader::with_capacity(4, &b"alpha\nbeta\ngamma"[..]);
let bytes = reader.read_bytes()?.unwrap();
assert_eq!(reader.read_view()?, Some(&b"beta"[..]));
let owned = reader.read_owned()?.unwrap();
assert_eq!(reader.read_view()?, None);
drop(reader);
assert_eq!(&bytes[..], b"alpha");
assert_eq!(owned, b"gamma");
# for capacity in [0, 1, 8] {
#     let mut reader = LineReader::with_capacity(capacity, &b"\n\n\nx\ny\nz"[..]);
#     assert!(reader.read_bytes()?.unwrap().is_empty());
#     assert_eq!(reader.read_view()?, Some(&b""[..]));
#     assert_eq!(reader.read_owned()?, Some(Vec::new()));
#     assert_eq!(&reader.read_bytes()?.unwrap()[..], b"x");
#     assert_eq!(reader.read_view()?, Some(&b"y"[..]));
#     assert_eq!(reader.read_owned()?, Some(b"z".to_vec()));
#     assert_eq!(reader.read_bytes()?, None);
#     assert_eq!(reader.read_view()?, None);
#     assert_eq!(reader.read_owned()?, None);
# }
# Ok::<(), io::Error>(())