Drop-in replacements for buffered I/O types in
These replacements retain the method names/signatures and implemented traits of their stdlib counterparts, making replacement as simple as swapping the import of the type:
- use std::io::BufReader; + use buf_redux::BufReader;
- use std::io::BufWriter; + use buf_redux::BufWriter;
- use std::io::LineWriter; + use buf_redux::LineWriter;
All replacement types provide methods to:
- Increase the capacity of the buffer
- Get the number of available bytes as well as the total capacity of the buffer
- Consume the wrapper without losing data
BufReader provides methods to:
- Access the buffer through an
&-reference without performing I/O
- Force unconditional reads into the buffer
- Get a
Readadapter which empties the buffer and then pulls from the inner reader directly
- Shuffle bytes down to the beginning of the buffer to make room for more reading
- Get inner reader and trimmed buffer with the remaining data
LineWriter provides methods to:
- Flush the buffer and unwrap the inner writer unconditionally.
- Get the inner writer and trimmed buffer with the unflushed data.
Tune the behavior of the buffer to your specific use-case using the types in the
BufReader's behavior by implementing the
ReaderPolicytrait or use an existing implementation like
MinBufferedto ensure the buffer always contains a minimum number of bytes (until the underlying reader is empty).
BufWriter's behavior by implementing the [
WriterPolicytrait] or use an existing implementation like
FlushOnto flush when a particular byte appears in the buffer (used to implement
The buffered types of this crate and their
std::io counterparts, by default, use
as their buffer types (
Buffer is included as well since it is used internally
by the other types in this crate).
When one of these types inserts bytes into its buffer, via
BufReader's case or
the entire buffer is provided to be read/written into and the number of bytes written is saved.
The read/written data then resides in the
[0 .. bytes_inserted] slice of the buffer.
When bytes are consumed from the buffer, via
the number of bytes consumed is added to the start of the slice such that the remaining
data resides in the
[bytes_consumed .. bytes_inserted] slice of the buffer.
std::io buffered types, and their counterparts in this crate with their default policies,
don't have to deal with partially filled buffers as
BufReader only reads when empty and
BufWriter only flushes when full.
However, because the replacements in this crate are capable of reading on-demand and flushing less than a full buffer, they can run out of room in their buffers to read/write data into even though there is technically free space, because this free space is at the head of the buffer where reading into it would cause the data in the buffer to become non-contiguous.
This isn't technically a problem as the buffer could operate like
std and return
both slices at once, but this would not fit all use-cases: the
Read::fill_buf() interface only
allows one slice to be returned at a time so the older data would need to be completely consumed
before the newer data can be returned;
BufWriter could support it as the
doesn't make an opinion on how the buffer works, but because the data would be non-contiguous
it would require two flushes to get it all, which could degrade performance.
The obvious solution, then, is to move the existing data down to the beginning of the buffer
when there is no more room at the end so that more reads/writes into the buffer can be issued.
This works, and may suit some use-cases where the amount of data left is small and thus copying
it would be inexpensive, but it is non-optimal. However, this option is provided
.make_room() methods, and is utilized by
Instead of moving data, however, it is also possible to use virtual-memory tricks to allocate a ringbuffer that loops around on itself in memory and thus is always contiguous, as described in the Wikipedia article on Ringbuffers.
This is the exact trick used by the
which is now provided as an optional feature
slice-deque exposed via the
with_capacity_ringbuf() constructors added to the buffered types here.
When a buffered type is constructed using one of these functions,
.make_room() is turned into
a no-op as consuming bytes from the head of the buffer simultaneously makes room at the tail.
However, this has some caveats:
It is only available on target platforms with virtual memory support, namely fully fledged OSes such as Windows and Unix-derivative platforms like Linux, OS X, BSD variants, etc.
The default capacity varies based on platform, and custom capacities are rounded up to a multiple of their minimum size, typically the page size of the platform. Windows' minimum size is comparably quite large (64 KiB) due to some legacy reasons, so this may be less optimal than the default capacity for a normal buffer (8 KiB) for some use-cases.
Due to the nature of the virtual-memory trick, the virtual address space the buffer allocates will be double its capacity. This means that your program will appear to use more memory than it would if it was using a normal buffer of the same capacity. The physical memory usage will be the same in both cases, but if address space is at a premium in your application (32-bit targets) then this may be a concern.
Types which can be used to tune the behavior of
A drop-in replacement for
A drop-in replacement for
A deque-like datastructure for managing bytes.
The error type for
A drop-in replacement for
Copy data between a