Struct Journal

pub struct Journal<B: Blob, E: Storage<B> + Metrics> { /* private fields */ }

Implementation of Journal storage.

Implementations

impl<B: Blob, E: Storage<B> + Metrics> Journal<B, E>

pub async fn init(context: E, cfg: Config) -> Result<Self, Error>

Initialize a new Journal instance.

All backing blobs are opened but not read during initialization. The replay method can be used to iterate over all items in the Journal.

pub async fn replay( &mut self, concurrency: usize, prefix: Option<u32>, ) -> Result<impl Stream<Item = Result<(u64, u32, u32, Bytes), Error>> + '_, Error>

Returns an unordered stream of all items in the journal.

Repair

If any corrupted data is found, the stream will return an error.

If any trailing data is found (i.e. misaligned entries), the journal will be truncated to the last valid item. For this reason, it is recommended to call replay before calling append (as data added to trailing bytes will fail checksum after restart).

Concurrency

The concurrency parameter controls how many blobs are replayed concurrently. This can dramatically speed up the replay process if the underlying storage supports concurrent reads across different blobs.

Prefix

If prefix is provided, the stream will only read up to prefix bytes of each item. Consequently, this means we will not compute a checksum of the entire data and it is up to the caller to deal with the consequences of this.

Reading prefix bytes and skipping ahead to a future location in a blob is the theoretically optimal way to read only what is required from storage, however, different storage implementations may take the opportunity to readahead past what is required (needlessly). If the underlying storage can be tuned for random access prior to invoking replay, it may lead to less IO.

pub async fn append(&mut self, section: u64, item: Bytes) -> Result<u32, Error>

Appends an item to Journal in a given section.

Warning

If there exist trailing bytes in the Blob of a particular section and replay is not called before this, it is likely that subsequent data added to the Blob will be considered corrupted (as the trailing bytes will fail the checksum verification). It is recommended to call replay before calling append to prevent this.

pub async fn get_prefix( &self, section: u64, offset: u32, prefix: u32, ) -> Result<Option<Bytes>, Error>

Retrieves the first prefix bytes of an item from Journal at a given section and offset.

This method bypasses the checksum verification and the caller is responsible for ensuring the integrity of any data read.

pub async fn get( &self, section: u64, offset: u32, exact: Option<u32>, ) -> Result<Option<Bytes>, Error>

Retrieves an item from Journal at a given section and offset.

If exact is provided, it is assumed the item is of size exact (which allows the item to be read in a single read). If exact is provided, the checksum of the data is still verified.

pub async fn sync(&self, section: u64) -> Result<(), Error>

Ensures that all data in a given section is synced to the underlying store.

If the section does not exist, no error will be returned.

pub async fn prune(&mut self, min: u64) -> Result<(), Error>

Prunes all sections less than min.

pub async fn close(self) -> Result<(), Error>

Closes all open sections.