Struct Journal 

pub struct Journal<E: Storage + Metrics, A: CodecFixedShared> { /* private fields */ }

Implementation of Journal storage.

This is implemented as a wrapper around SegmentedJournal that provides position-based access where positions are automatically mapped to (section, position_in_section) pairs.

Repair

Like sqlite and rocksdb, the first invalid data read will be considered the new end of the journal (and the underlying blob will be truncated to the last valid item). Repair is performed by the underlying SegmentedJournal during init.

Implementations

impl<E: Storage + Metrics, A: CodecFixedShared> Journal<E, A>

pub const CHUNK_SIZE: usize = SegmentedJournal<E, A>::CHUNK_SIZE

Size of each entry in bytes.

pub const CHUNK_SIZE_U64: u64

Size of each entry in bytes (as u64).

pub async fn init(context: E, cfg: Config) -> Result<Self, Error>

Initialize a new Journal instance.

All backing blobs are opened but not read during initialization. The replay method can be used to iterate over all items in the Journal.

pub async fn init_at_size( context: E, cfg: Config, size: u64, ) -> Result<Self, Error>

Initialize a new Journal instance in a pruned state at a given size.

This is used for state sync to create a journal that appears to have had size items appended and then pruned up to that point.

Arguments

• context - The storage context
• cfg - Configuration for the journal
• size - The number of operations that have been “pruned”

Behavior

• Creates only the tail blob at the section that would contain position size-1
• The items in the tail blob before size are filled with zeros (dummy data)
• oldest_retained_pos() returns the start of the tail section, matching behavior if the journal were reopened normally
• Positions within the tail section but before size contain dummy zero data

Invariants

• The directory given by cfg.partition should be empty

pub async fn sync(&mut self) -> Result<(), Error>

Sync any pending updates to disk.

Only the tail section can have pending updates since historical sections are synced when they become full.

pub const fn size(&self) -> u64

Return the total number of items in the journal, irrespective of pruning. The next value appended to the journal will be at this position.

pub async fn append(&mut self, item: A) -> Result<u64, Error>

Append a new item to the journal. Return the item’s position in the journal, or error if the operation fails.

pub async fn rewind(&mut self, size: u64) -> Result<(), Error>

Rewind the journal to the given size. Returns Error::InvalidRewind if the rewind point precedes the oldest retained element point. The journal is not synced after rewinding.

Warnings

• This operation is not guaranteed to survive restarts until sync is called.
• This operation is not atomic, but it will always leave the journal in a consistent state in the event of failure since blobs are always removed from newest to oldest.

pub fn oldest_retained_pos(&self) -> Option<u64>

Return the position of the oldest item in the journal that remains readable.

Note that this value could be older than the min_item_pos last passed to prune.

pub fn pruning_boundary(&self) -> u64

Return the location before which all items have been pruned.

pub async fn read(&self, pos: u64) -> Result<A, Error>

Read the item at position pos in the journal.

Errors

• Error::ItemPruned if the item at position pos is pruned.
• Error::ItemOutOfRange if the item at position pos does not exist.

pub async fn replay( &self, buffer: NonZeroUsize, start_pos: u64, ) -> Result<impl Stream<Item = Result<(u64, A), Error>> + '_, Error>

Returns an ordered stream of all items in the journal with position >= start_pos.

Integrity

If any corrupted data is found, or if any non-tail section has fewer items than items_per_blob, the stream will return an error.

pub async fn prune(&mut self, min_item_pos: u64) -> Result<bool, Error>

Allow the journal to prune items older than min_item_pos. The journal may not prune all such items in order to preserve blob boundaries, but the amount of such items will always be less than the configured number of items per blob. Returns true if any items were pruned.

Note that this operation may NOT be atomic, however it’s guaranteed not to leave gaps in the event of failure as items are always pruned in order from oldest to newest.

pub async fn destroy(self) -> Result<(), Error>

Remove any underlying blobs created by the journal.

Trait Implementations

impl<E: Storage + Metrics, A: CodecFixedShared> Contiguous for Journal<E, A>

type Item = A

The type of items stored in the journal.

fn size(&self) -> u64

Return the total number of items that have been appended to the journal.

fn oldest_retained_pos(&self) -> Option<u64>

Return the position of the oldest item still retained in the journal.

fn pruning_boundary(&self) -> u64

Return the location before which all items have been pruned.

async fn replay( &self, start_pos: u64, buffer: NonZeroUsize, ) -> Result<impl Stream<Item = Result<(u64, Self::Item), Error>> + '_, Error>

Return a stream of all items in the journal starting from start_pos.

async fn read(&self, position: u64) -> Result<Self::Item, Error>

Read the item at the given position.

impl<E: Storage + Metrics, A: CodecFixedShared> MutableContiguous for Journal<E, A>

async fn append(&mut self, item: Self::Item) -> Result<u64, Error>

Append a new item to the journal, returning its position.

async fn prune(&mut self, min_position: u64) -> Result<bool, Error>

Prune items at positions strictly less than min_position.

async fn rewind(&mut self, size: u64) -> Result<(), Error>

Rewind the journal to the given size, discarding items from the end.

fn rewind_to<'a, P>( &'a mut self, predicate: P, ) -> impl Future<Output = Result<u64, Error>> + Send + 'a

where P: FnMut(&Self::Item) -> bool + Send + 'a,

Rewinds the journal to the last item matching predicate. If no item matches, the journal is rewound to the pruning boundary, discarding all unpruned items.

impl<E: Storage + Metrics, A: CodecFixedShared> Persistable for Journal<E, A>

type Error = Error

The error type returned when there is a failure from the underlying storage system.

async fn commit(&mut self) -> Result<(), Error>

Durably persist the structure, guaranteeing the current state will survive a crash.

async fn sync(&mut self) -> Result<(), Error>

Durably persist the structure, guaranteeing the current state will survive a crash, and that no recovery will be needed on startup.

async fn destroy(self) -> Result<(), Error>

Destroy the structure, removing all associated storage.