Module block 

A stream over a fixed number of fixed-sized blocks.

The primary problem that Stream solves is small and variable-sized record writes over larger fixed-sized blocks, described by Blocks trait. The rest is more or less a consequence of that.

This implementation uses each block to hold metadata. This adds an extra overhead of in-memory copy during writes, yet provides better integrity, allows for more efficient navigation, and keeps the IO part as basic as possible, which is easier to reason about. The layout can be used for a binary search, or may be even parallel computations over data, as it is possible to find the start of a write for any given block.

This abstraction is generic enough to work over block devices without a filesystem, or with conventional files.

Things to note

The whole stream is loaded as a contiguous memory. There is no page caching.

Metadata consumes 48 bytes from every block, consider this when calculating the effective capacity of the stream.

I haven’t thought of using this with large records much. It may not be the best choice for that scenario and it may be more wise to use files or another more specialized structure after a certain point, yet this stream can still hold the metadata. What is large specifically? Depends on the hardware and block size, but just to throw some random number, everything larger than 8M per record should be considered carefully.

Structs

LockedStream

A locked Stream. Returned by Stream::try_lock or Stream::lock.

Stream

A fixed-size stream implemented on top of Blocks with serial writes and wait-free read access.

Enums

Inconsistency

An error status returned during Stream::verify call for every block.

StreamError

An error specific to the Stream.

Traits

Blocks

Something that can be subdivided into blocks, each being a single unit of operation.