Module fixed

An append-only log for storing fixed length items on disk.

In addition to replay, stored items can be fetched directly by their position in the journal, where position is defined as the item’s order of insertion starting from 0, unaffected by pruning.

See the variable crate for a journal that supports variable length items.

Format

Data stored in a fixed::Journal is persisted in one of many Blobs within a caller-provided partition. Each Blob contains a configurable maximum of items_per_blob, with each item followed by its checksum (CRC32):

+--------+-----------+--------+-----------+--------+----------+-------------+
| item_0 | C(Item_0) | item_1 | C(Item_1) |   ...  | item_n-1 | C(Item_n-1) |
+--------+-----------+--------+----0------+--------+----------+-------------+

n = config.items_per_blob, C = CRC32

The most recent blob may not necessarily be full, in which case it will contain fewer than the maximum number of items.

A fetched or replayed item’s checksum is always computed and checked against the stored value before it is returned. If the checksums do not match, an error is returned instead.

Open Blobs

All Blobs in a given partition are kept open during the lifetime of Journal. You can limit the number of open blobs by using a higher number of items_per_blob or pruning old items.

Sync

Data written to Journal may not be immediately persisted to Storage. It is up to the caller to determine when to force pending data to be written to Storage using the sync method. When calling close, all pending data is automatically synced and any open blobs are closed.

Pruning

Old data can be pruned from Journal by calling the prune method which will remove all blobs consisting entirely of values older than the given item.

Replay

The replay method iterates over multiple blobs concurrently to support fast reading of all unpruned items into memory.

Structs

Config

Configuration for Journal storage.

Journal

Implementation of Journal storage.