Struct fixed_buffer::FixedBuf

pub struct FixedBuf<T> { /* fields omitted */ }

FixedBuf is a fixed-length byte buffer. You can write bytes to it and then read them back.

It implements tokio's AsyncRead and AsyncWrite traits.

Use read_delimited to read lines and other delimited messages. This works like tokio::io::AsyncBufReadExt::read_until, but uses a fixed sized buffer so network peers cannot OOM the process.

It is not a circular buffer. Call shift periodically to move unread bytes to the front of the buffer.

Use new to create FixedBuf<[u8; N]>, FixedBuf<Box<[u8]>>, and FixedBuf<&mut [u8]> structs.

Note that FixedBuf<Box<[u8]>> uses less memory than Box<FixedBuf<[u8; N]>>. See new for details.

Implementations

impl<T> FixedBuf<T>

pub const fn new(mem: T) -> Self

Makes a new empty buffer, consuming or borrowing mem and using it as the internal memory array.

Creates FixedBuf<[u8; N]>, FixedBuf<Box<[u8]>>, and FixedBuf<&mut [u8]> structs.

This function is the inverse of into_inner.

FixedBuf<&mut [u8]> uses borrowed memory. Create one like this:

let mut mem = [0u8; 42];
let mut buf: FixedBuf<&mut [u8]> = FixedBuf::new(&mut mem);

FixedBuf<[u8; N]> can live on the stack. Be careful of stack overflows! Create one like this:

let mut buf: FixedBuf<[u8; 42]> = FixedBuf::new([0u8; 42]);

FixedBuf<Box<[u8]>> stores its memory block on the heap. Box supports coercion of Box<[T; N]> to Box<[T]>. Use that to create a boxed buffer:

let mut buf: FixedBuf<Box<[u8]>> = FixedBuf::new(Box::new([0u8; 42]));
// Your editor may incorrectly report "mismatched types [E0308]".

Note that FixedBuf<Box<[u8]>> is 10-25% more memory efficient than Box<FixedBuf<[u8; N]>>. Explanation:

Standard heaps allocate memory in blocks. The block sizes are powers of two and some intervening sizes. For example, jemalloc's block sizes are 8, 16, 32, 48, 64, 80, 96, 112, 128, 160, 192, 224, 256 bytes, and so on. Thus jemalloc uses 160 bytes to store a 129 byte value.

Every FixedBuf<[u8; N]> contains two usize index values in addition to its buffer memory. If you create a buffer with a power-of-two size, the struct is always a few bytes larger than a power of two, and takes up the next larger block size on the heap. For example, in a 64-bit program using jemalloc, Box<FixedBuf<[u8; 128]>> uses 128 + 8 + 8 = 144 bytes, and gets stored in a 160 byte block, wasting an extra 11% of memory.

By comparison, FixedBuf<Box<[u8]>> keeps the buffer memory separate from the index values and therefore wastes no memory. This is because Box<[u8; 128]> uses exactly 128-bytes on the heap.

Run the program examples/box_benchmark.rs to see the memory usage difference.

pub fn into_inner(self) -> T

Drops the struct and returns its internal array.

This function is the inverse of new.

pub fn len(&self) -> usize

Returns the number of unread bytes in the buffer.

Example:

let mut buf: FixedBuf<[u8; 16]> = FixedBuf::new([0u8; 16]);
assert_eq!(0, buf.len());
buf.write_str("abc");
assert_eq!(3, buf.len());
buf.read_bytes(2);
assert_eq!(1, buf.len());
buf.shift();
assert_eq!(1, buf.len());
buf.read_all();
assert_eq!(0, buf.len());

pub fn is_empty(&self) -> bool

Returns true if there are unread bytes in the buffer.

Example:

let mut buf: FixedBuf<[u8; 16]> = FixedBuf::new([0u8; 16]);
assert!(buf.is_empty());
buf.write_str("abc").unwrap();
assert!(!buf.is_empty());
buf.read_all();
assert!(buf.is_empty());

pub fn clear(&mut self)

Discards all data in the buffer.

impl<T: AsRef<[u8]>> FixedBuf<T>

pub fn filled(mem: T) -> Self

Makes a new full buffer, consuming or borrowing mem and using it as the internal memory array. Reading the buffer will return the bytes in mem.

You can write to the returned buf if mem implements AsMut<[u8].

For details, see [new].

Examples:

// Readable, not writable:
let mut buf1 = FixedBuf::filled(b"abc");

// Readable and writable:
let mut buf2 = FixedBuf::filled([0u8; 42]);
let mut buf3: FixedBuf<[u8; 42]> = FixedBuf::filled([0u8; 42]);
let mut buf4: FixedBuf<Box<[u8]>> = FixedBuf::filled(Box::new([0u8; 42]));
// Your editor may incorrectly report "mismatched types [E0308]" --^

pub fn capacity(&self) -> usize

Returns the maximum number of bytes that can be stored in the buffer.

Example:

let mut buf: FixedBuf<[u8; 16]> = FixedBuf::new([0u8; 16]);
assert_eq!(16, buf.capacity());
buf.write_str("abc").unwrap();
assert_eq!(16, buf.capacity());

pub fn escape_ascii(&self) -> String

Copies all readable bytes to a string. Includes printable ASCII characters as-is. Converts non-printable characters to strings like "\n" and "\x19".

Leaves the buffer unchanged.

Uses std::ascii::escape_default internally to escape each byte.

This function is useful for printing byte slices to logs and comparing byte slices in tests.

Example test:

use fixed_buffer::FixedBuf;
let mut buf = FixedBuf::new([0u8; 16]);
buf.write_str("abc");
buf.write_str("€");
assert_eq!("abc\\xe2\\x82\\xac", buf.escape_ascii());

pub fn readable(&self) -> &[u8]

Returns the slice of readable bytes in the buffer. After processing some bytes from the front of the slice, call read to consume the bytes.

This is a low-level method. You probably want to use read, std::io::Read::read, and tokio::io::AsyncReadExt::read instead.

Example:

fn try_process_record(b: &[u8]) -> Result<usize, Error> {
    if b.len() < 2 {
        return Ok(0);
    }
    if b.starts_with("ab".as_bytes()) {
        println!("found record");
        Ok(2)
    } else {
        Err(Error::new(ErrorKind::InvalidData, "bad record"))
    }
}

async fn read_and_process<R: tokio::io::AsyncRead + Unpin>(mut input: R)
    -> Result<(), Error> {
    let mut buf: FixedBuf<[u8; 1024]> = FixedBuf::new([0; 1024]);
    loop {
        // Read a chunk into the buffer.
        let mut writable = buf.writable()
            .ok_or(Error::new(ErrorKind::InvalidData, "record too long, buffer full"))?;
        let bytes_written = AsyncReadExt::read(&mut input, &mut writable).await?;
        if bytes_written == 0 {
            return if buf.len() == 0 {
                Ok(())  // EOF at record boundary
            } else {
                // EOF in the middle of a record
                Err(Error::from(ErrorKind::UnexpectedEof))
            };
        }
        buf.wrote(bytes_written);

        // Process records in the buffer.
        loop {
            let bytes_read = try_process_record(buf.readable())?;
            if bytes_read == 0 {
                break;
            }
            buf.read_bytes(bytes_read);
        }
        // Shift data in the buffer to free up space at the end for writing.
        buf.shift();
    }
}

read_and_process(std::io::Cursor::new(b"")).await.unwrap();
read_and_process(std::io::Cursor::new(b"abab")).await.unwrap();
assert_eq!(
    std::io::ErrorKind::UnexpectedEof,
    read_and_process(std::io::Cursor::new(b"aba")).await.unwrap_err().kind()
);

pub fn read_bytes(&mut self, num_bytes: usize) -> &[u8]

Read bytes from the buffer.

Panics if the buffer does not contain enough bytes.

pub fn read_all(&mut self) -> &[u8]

Read all the bytes from the buffer. The buffer becomes empty and subsequent writes can fill the whole buffer.

pub fn read_and_copy_bytes(&mut self, dest: &mut [u8]) -> Result<usize>

Reads bytes from the buffer and copies them into dest.

Returns the number of bytes copied.

Returns Ok(0) when the buffer is empty or dest is zero-length.

impl<T: AsRef<[u8]> + AsMut<[u8]>> FixedBuf<T>

pub async fn read_delimited<R, '_, '_, '_>(
    &'_ mut self,
    input: R,
    delim: &'_ [u8]
) -> Result<Option<&'_ [u8]>> where
    R: AsyncRead + Unpin + Send, 

Reads from a tokio::io::AsyncRead into the buffer until it finds delim. Returns the slice up until delim. Consumes the returned bytes and delim. Leaves unused bytes in the buffer.

If the buffer already contains delim, returns the data immediately without reading from input.

If the buffer does not already contain delim, calls shift before reading from input.

Returns Err(Error(InvalidData,_)) if the buffer fills up before delim is found.

Returns Err(Error(UnexpectedEof,_)) when input returns some bytes without delim at the end and then closes (EOF).

Demo:

let mut buf: FixedBuf<[u8; 32]> = FixedBuf::new([0u8; 32]);
let mut input = std::io::Cursor::new(b"aaa\nbbb\n\nccc\n");
assert_eq!("aaa", escape_ascii(buf.read_delimited(&mut input, b"\n").await.unwrap().unwrap()));
assert_eq!("bbb", escape_ascii(buf.read_delimited(&mut input, b"\n").await.unwrap().unwrap()));
assert_eq!("",    escape_ascii(buf.read_delimited(&mut input, b"\n").await.unwrap().unwrap()));
assert_eq!("ccc", escape_ascii(buf.read_delimited(&mut input, b"\n").await.unwrap().unwrap()));
assert_eq!(None,  buf.read_delimited(&mut input, b"\n").await.unwrap());

Example usage:

let (mut input, mut output) = tcp_stream.split();
let mut buf: FixedBuf<[u8; 1024]> = FixedBuf::new([0; 1024]);
loop {
    // Read a line and leave leftover bytes in `buf`.
    let line_bytes = match buf.read_delimited(&mut input, b"\n").await? {
        Some(line_bytes) => line_bytes,
        None => return Ok(()),
    };
    let request = Request::parse(line_bytes)?;
    // Read any request payload from `buf` + `TcpStream`.
    let payload_reader = tokio::io::AsyncReadExt::chain(&mut buf, &mut input);
    handle_request(&mut output, payload_reader, request).await?;
}

impl<T: AsMut<[u8]>> FixedBuf<T>

pub fn write_str(&mut self, s: &str) -> Result<()>

Writes s into the buffer, after any unread bytes.

Returns Err if the buffer doesn't have enough free space at the end for the whole string.

See shift.

Example:

let mut buf: FixedBuf<[u8; 8]> = FixedBuf::new([0u8; 8]);
buf.write_str("123").unwrap();
buf.write_str("456").unwrap();
assert_eq!("1234", escape_ascii(buf.read_bytes(4)));
buf.write_str("78").unwrap();
buf.write_str("9").unwrap_err();  // End of buffer is full.

pub fn write_bytes(&mut self, data: &[u8]) -> Result<usize>

Try to write data into the buffer, after any unread bytes.

Returns Ok(data.len()) if it wrote all of the bytes.

Returns Err if the buffer doesn't have enough free space at the end for all of the bytes.

See shift.

Example:

let mut buf: FixedBuf<[u8; 8]> = FixedBuf::new([0u8; 8]);
assert_eq!(3 as usize, buf.write_bytes("123".as_bytes()).unwrap());
assert_eq!(3 as usize, buf.write_bytes("456".as_bytes()).unwrap());
assert_eq!("1234", escape_ascii(buf.read_bytes(4)));
assert_eq!(2 as usize, buf.write_bytes("78".as_bytes()).unwrap());  // Fills buffer.
buf.write_bytes("9".as_bytes()).unwrap_err();  // Error, buffer is full.

pub fn writable(&mut self) -> Option<&mut [u8]>

Returns the writable part of the buffer.

To use this, first modify bytes at the beginning of the slice. Then call wrote(usize) to commit those bytes into the buffer and make them available for reading.

Returns None when the end of the buffer is full. See shift.

This is a low-level method. You probably want to use std::io::Write::write and tokio::io::AsyncWriteExt::write instead.

Example:

let mut buf: FixedBuf<[u8; 8]> = FixedBuf::new([0u8; 8]);
buf.writable().unwrap()[0] = 'a' as u8;
buf.writable().unwrap()[1] = 'b' as u8;
buf.writable().unwrap()[2] = 'c' as u8;
buf.wrote(3);
assert_eq!("abc", escape_ascii(buf.read_bytes(3)));

pub fn wrote(&mut self, num_bytes: usize)

Commit bytes into the buffer. Call this after writing to the front of the writable slice.

This is a low-level method.

Panics if writable() is not large enough.

See example in writable().

pub fn shift(&mut self)

Recovers buffer space.

The buffer is not circular. After you read bytes, the space at the beginning of the buffer is unused. Call this method to move unread data to the beginning of the buffer and recover the space. This makes the free space available for writes, which go at the end of the buffer.

For an example, see readable.

Trait Implementations

impl<T: AsRef<[u8]> + Unpin> AsyncRead for FixedBuf<T>

pub fn poll_read(
    self: Pin<&mut Self>,
    _cx: &mut Context<'_>,
    buf: &mut ReadBuf<'_>
) -> Poll<Result<(), Error>>

Attempts to read from the AsyncRead into buf.

impl<T: AsMut<[u8]>> AsyncWrite for FixedBuf<T>

pub fn poll_write(
    self: Pin<&mut Self>,
    _cx: &mut Context<'_>,
    buf: &[u8]
) -> Poll<Result<usize, Error>>

Attempt to write bytes from buf into the object.

pub fn poll_flush(
    self: Pin<&mut Self>,
    _cx: &mut Context<'_>
) -> Poll<Result<(), Error>>

Attempts to flush the object, ensuring that any buffered data reach their destination.

pub fn poll_shutdown(
    self: Pin<&mut Self>,
    _cx: &mut Context<'_>
) -> Poll<Result<(), Error>>

Initiates or attempts to shut down this writer, returning success when the I/O connection has completely shut down.

impl<T: Clone> Clone for FixedBuf<T>

pub fn clone(&self) -> FixedBuf<T>

Returns a copy of the value.

pub fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Copy> Copy for FixedBuf<T>

impl<T: AsRef<[u8]>> Debug for FixedBuf<T>

pub fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: Default> Default for FixedBuf<T>

pub fn default() -> FixedBuf<T>

Returns the "default value" for a type.

impl<T: Eq> Eq for FixedBuf<T>

impl<T: Hash> Hash for FixedBuf<T>

pub fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

pub fn hash_slice<H>(data: &[Self], state: &mut H) where
    H: Hasher, 

Feeds a slice of this type into the given Hasher.

impl<T: PartialEq> PartialEq<FixedBuf<T>> for FixedBuf<T>

pub fn eq(&self, other: &FixedBuf<T>) -> bool

This method tests for self and other values to be equal, and is used by ==.

pub fn ne(&self, other: &FixedBuf<T>) -> bool

This method tests for !=.

impl<T: AsRef<[u8]>> Read for FixedBuf<T>

pub fn read(&mut self, buf: &mut [u8]) -> Result<usize>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

pub fn read_vectored(
    &mut self,
    bufs: &mut [IoSliceMut<'_>]
) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

pub fn is_read_vectored(&self) -> bool

🔬 This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

pub unsafe fn initializer(&self) -> Initializer

🔬 This is a nightly-only experimental API. (read_initializer)

Determines if this Reader can work with buffers of uninitialized memory.

pub fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Read all bytes until EOF in this source, placing them into buf.

pub fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Read all bytes until EOF in this source, appending them to buf.

pub fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Read the exact number of bytes required to fill buf.

pub fn by_ref(&mut self) -> &mut Self

Creates a "by reference" adaptor for this instance of Read.

pub fn bytes(self) -> Bytes<Self>

Transforms this Read instance to an Iterator over its bytes.

pub fn chain<R>(self, next: R) -> Chain<Self, R> where
    R: Read, 

Creates an adaptor which will chain this stream with another.

pub fn take(self, limit: u64) -> Take<Self>

Creates an adaptor which will read at most limit bytes from it.

impl<T> StructuralEq for FixedBuf<T>

impl<T> StructuralPartialEq for FixedBuf<T>

impl<T> Unpin for FixedBuf<T>

impl<T: AsMut<[u8]>> Write for FixedBuf<T>

pub fn write(&mut self, data: &[u8]) -> Result<usize>

Write a buffer into this writer, returning how many bytes were written.

pub fn flush(&mut self) -> Result<()>

Flush this output stream, ensuring that all intermediately buffered contents reach their destination.

pub fn write_vectored(&mut self, bufs: &[IoSlice<'_>]) -> Result<usize, Error>

Like write, except that it writes from a slice of buffers.

pub fn is_write_vectored(&self) -> bool

🔬 This is a nightly-only experimental API. (can_vector)

Determines if this Writer has an efficient write_vectored implementation.

pub fn write_all(&mut self, buf: &[u8]) -> Result<(), Error>

Attempts to write an entire buffer into this writer.

pub fn write_all_vectored(
    &mut self,
    bufs: &mut [IoSlice<'_>]
) -> Result<(), Error>

🔬 This is a nightly-only experimental API. (write_all_vectored)

Attempts to write multiple buffers into this writer.

pub fn write_fmt(&mut self, fmt: Arguments<'_>) -> Result<(), Error>

Writes a formatted string into this writer, returning any error encountered.

pub fn by_ref(&mut self) -> &mut Self

Creates a "by reference" adaptor for this instance of Write.