Struct bytes_utils::SegmentedBuf

pub struct SegmentedBuf<B> { /* private fields */ }

A concatenation of multiple buffers into a large one, without copying the bytes over.

Note that this doesn’t provide a continuous slice view into them, it is split into the segments of the original smaller buffers.

This variants drop the inner buffers as they are exhausted and new ones can be added. But it internally keeps a VecDeque, therefore needs a heap allocation. If you don’t need the extending behaviour, but want to avoid the allocation, the SegmentedSlice can be used instead.

Why

This can be used, for example, if data of unknown length is coming over the network (for example, the bodies in hyper act a bit like this, it returns a stream of Bytes buffers). One might want to accumulate the whole body before acting on it, possibly by parsing it through serde or prost. Options would include:

• Have a Vec<u8> and extend it with each chunk. This needlessly copy the bytes every time and reallocates if the vector grows too large.
• Repeatedly use chain, but this changes the type of the whole buffer, therefore needs to be boxed.
• Use hyper::body::aggregate to create a Buf implementation that concatenates all of them together, but lacks any kind of flexibility (like protecting against loading too much data into memory).

This type allows for concatenating multiple buffers, either all at once, or by incrementally pushing more buffers to the end.

Heterogeneous buffers

This expects all the buffers are of the same type. If different-typed buffers are needed, one needs to use dynamic dispatch, either something like SegmentedBuf<Box<Buf>> or SegmentedBuf<&mut Buf>.

Example

let mut buf = SegmentedBuf::new();
buf.push(Bytes::from("Hello"));
buf.push(Bytes::from(" "));
buf.push(Bytes::from("World"));

assert_eq!(3, buf.segments());
assert_eq!(11, buf.remaining());
assert_eq!(b"Hello", buf.chunk());

let mut out = String::new();
buf.reader().read_to_string(&mut out).expect("Doesn't cause IO errors");
assert_eq!("Hello World", out);

FIFO behaviour

The buffers are dropped once their data are completely consumed. Additionally, it is possible to add more buffers to the end, even while some of the previous buffers were partially or fully consumed. That makes it usable as kind of a queue (that operates on the buffers, not individual bytes).

let mut buf = SegmentedBuf::new();
buf.push(Bytes::from("Hello"));
assert_eq!(1, buf.segments());

let mut out = [0; 3];
buf.copy_to_slice(&mut out);
assert_eq!(&out, b"Hel");
assert_eq!(2, buf.remaining());
assert_eq!(1, buf.segments());

buf.push(Bytes::from("World"));
assert_eq!(7, buf.remaining());
assert_eq!(2, buf.segments());

buf.copy_to_slice(&mut out);
assert_eq!(&out, b"loW");
assert_eq!(4, buf.remaining());
assert_eq!(1, buf.segments());

Optimizations

The copy_to_bytes method tries to avoid copies by delegating into the underlying buffer if possible (if the whole request can be fulfilled using only a single buffer). If that one is optimized (for example, the Bytes returns a shared instance instead of making a copy), the copying is avoided. If the request is across a buffer boundary, a copy is made.

The chunks_vectored will properly output as many slices as possible, not just 1 as the default implementation does.

Implementations

impl<B> SegmentedBuf<B>

pub fn new() -> Self

Creates a new empty instance.

The instance can be pushed or extended later.

Alternatively, one may create it directly from an iterator, a Vec or a VecDeque of buffers.

pub fn into_inner(self) -> VecDeque<B>

Returns the yet unconsumed sequence of buffers.

pub fn segments(&self) -> usize

Returns the number of segments (buffers) this contains.

impl<B: Buf> SegmentedBuf<B>

pub fn push(&mut self, buf: B)

Extends the buffer by another segment.

The newly added segment is added to the end of the buffer (the buffer works as a FIFO).

Trait Implementations

impl<B: Buf> Buf for SegmentedBuf<B>

fn remaining(&self) -> usize

Returns the number of bytes between the current position and the end of the buffer.

fn chunk(&self) -> &[u8]

Returns a slice starting at the current position and of length between 0 and Buf::remaining(). Note that this can return shorter slice (this allows non-continuous internal representation).

fn advance(&mut self, cnt: usize)

Advance the internal cursor of the Buf

fn copy_to_bytes(&mut self, len: usize) -> Bytes

Consumes len bytes inside self and returns new instance of Bytes with this data.

fn chunks_vectored<'a>(&'a self, dst: &mut [IoSlice<'a>]) -> usize

Fills dst with potentially multiple slices starting at self’s current position.

fn has_remaining(&self) -> bool

Returns true if there are any more bytes to consume

fn copy_to_slice(&mut self, dst: &mut [u8])

Copies bytes from self into dst.

fn get_u8(&mut self) -> u8

Gets an unsigned 8 bit integer from self.

fn get_i8(&mut self) -> i8

Gets a signed 8 bit integer from self.

fn get_u16(&mut self) -> u16

Gets an unsigned 16 bit integer from self in big-endian byte order.

fn get_u16_le(&mut self) -> u16

Gets an unsigned 16 bit integer from self in little-endian byte order.

fn get_i16(&mut self) -> i16

Gets a signed 16 bit integer from self in big-endian byte order.

fn get_i16_le(&mut self) -> i16

Gets a signed 16 bit integer from self in little-endian byte order.

fn get_u32(&mut self) -> u32

Gets an unsigned 32 bit integer from self in the big-endian byte order.

fn get_u32_le(&mut self) -> u32

Gets an unsigned 32 bit integer from self in the little-endian byte order.

fn get_i32(&mut self) -> i32

Gets a signed 32 bit integer from self in big-endian byte order.

fn get_i32_le(&mut self) -> i32

Gets a signed 32 bit integer from self in little-endian byte order.

fn get_u64(&mut self) -> u64

Gets an unsigned 64 bit integer from self in big-endian byte order.

fn get_u64_le(&mut self) -> u64

Gets an unsigned 64 bit integer from self in little-endian byte order.

fn get_i64(&mut self) -> i64

Gets a signed 64 bit integer from self in big-endian byte order.

fn get_i64_le(&mut self) -> i64

Gets a signed 64 bit integer from self in little-endian byte order.

fn get_u128(&mut self) -> u128

Gets an unsigned 128 bit integer from self in big-endian byte order.

fn get_u128_le(&mut self) -> u128

Gets an unsigned 128 bit integer from self in little-endian byte order.

fn get_i128(&mut self) -> i128

Gets a signed 128 bit integer from self in big-endian byte order.

fn get_i128_le(&mut self) -> i128

Gets a signed 128 bit integer from self in little-endian byte order.

fn get_uint(&mut self, nbytes: usize) -> u64

Gets an unsigned n-byte integer from self in big-endian byte order.

fn get_uint_le(&mut self, nbytes: usize) -> u64

Gets an unsigned n-byte integer from self in little-endian byte order.

fn get_int(&mut self, nbytes: usize) -> i64

Gets a signed n-byte integer from self in big-endian byte order.

fn get_int_le(&mut self, nbytes: usize) -> i64

Gets a signed n-byte integer from self in little-endian byte order.

fn get_f32(&mut self) -> f32

Gets an IEEE754 single-precision (4 bytes) floating point number from self in big-endian byte order.

fn get_f32_le(&mut self) -> f32

Gets an IEEE754 single-precision (4 bytes) floating point number from self in little-endian byte order.

fn get_f64(&mut self) -> f64

Gets an IEEE754 double-precision (8 bytes) floating point number from self in big-endian byte order.

fn get_f64_le(&mut self) -> f64

Gets an IEEE754 double-precision (8 bytes) floating point number from self in little-endian byte order.

fn take(self, limit: usize) -> Take<Self>

Creates an adaptor which will read at most limit bytes from self.

fn chain<U>(self, next: U) -> Chain<Self, U> where
    U: Buf, 

Creates an adaptor which will chain this buffer with another.

fn reader(self) -> Reader<Self>

Creates an adaptor which implements the Read trait for self.

impl<B: Clone> Clone for SegmentedBuf<B>

fn clone(&self) -> SegmentedBuf<B>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<B: Debug> Debug for SegmentedBuf<B>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<B> Default for SegmentedBuf<B>

fn default() -> Self

Returns the “default value” for a type.

impl<B: Buf> Extend<B> for SegmentedBuf<B>

fn extend<T: IntoIterator<Item = B>>(&mut self, iter: T)

Extends a collection with the contents of an iterator.

fn extend_one(&mut self, item: A)

🔬This is a nightly-only experimental API. (extend_one)

Extends a collection with exactly one element.

fn extend_reserve(&mut self, additional: usize)

🔬This is a nightly-only experimental API. (extend_one)

Reserves capacity in a collection for the given number of additional elements.

impl<B> From<SegmentedBuf<B>> for VecDeque<B>

fn from(me: SegmentedBuf<B>) -> Self

Converts to this type from the input type.

impl<B: Buf> From<Vec<B, Global>> for SegmentedBuf<B>

fn from(bufs: Vec<B>) -> Self

Converts to this type from the input type.

impl<B: Buf> From<VecDeque<B, Global>> for SegmentedBuf<B>

fn from(bufs: VecDeque<B>) -> Self

Converts to this type from the input type.

impl<B: Buf> FromIterator<B> for SegmentedBuf<B>

fn from_iter<T: IntoIterator<Item = B>>(iter: T) -> Self

Creates a value from an iterator.