Struct Commitlog 

pub struct Commitlog<T> { /* private fields */ }

The canonical commitlog, backed by on-disk log files.

Records in the log are of type T, which canonically is instantiated to payload::Txdata.

Implementations

impl<T> Commitlog<T>

pub fn open(root: CommitLogDir, opts: Options) -> Result<Self>

Open the log at root directory root with Options.

The root directory must already exist.

Note that opening a commitlog involves I/O: some consistency checks are performed, and the next writing position is determined.

This is only necessary when opening the commitlog for writing. See the free-standing functions in this module for how to traverse a read-only commitlog.

pub fn max_committed_offset(&self) -> Option<u64>

Determine the maximum transaction offset considered durable.

The offset is None if the log hasn’t been flushed to disk yet.

pub fn min_committed_offset(&self) -> Option<u64>

Determine the minimum transaction offset in the log.

The offset is None if the log hasn’t been flushed to disk yet.

pub fn epoch(&self) -> u64

Get the current epoch.

See also: Commit::epoch.

pub fn set_epoch(&self, epoch: u64) -> Result<Option<u64>>

Update the current epoch.

Does nothing if the given epoch is equal to the current epoch. Otherwise flushes outstanding transactions to disk (equivalent to Self::flush) before updating the epoch.

Returns the maximum transaction offset written to disk. The offset is None if the log is empty and no data was pending to be flushed.

Errors

If epoch is smaller than the current epoch, an error of kind io::ErrorKind::InvalidInput is returned.

Errors from the implicit flush are propagated.

pub fn sync(&self) -> Option<u64>

Sync all OS-buffered writes to disk.

Note that this does not write outstanding records to disk. Use Self::flush_and_sync or call Self::flush prior to this method to ensure all data is on disk.

Returns the maximum transaction offset which is considered durable after this method returns successfully. The offset is None if the log hasn’t been flushed to disk yet.

Panics

This method panics if syncing fails irrecoverably.

pub fn flush(&self) -> Result<Option<u64>>

Write all outstanding transaction records to disk.

Note that this does not force the OS to sync the data to disk. Use Self::flush_and_sync or call Self::sync after this method to ensure all data is on disk.

Returns the maximum transaction offset written to disk. The offset is None if the log is empty and no data was pending to be flushed.

Repeatedly calling this method may return the same value.

pub fn flush_and_sync(&self) -> Result<Option<u64>>

Write all outstanding transaction records to disk and flush OS buffers.

Equivalent to calling Self::flush followed by Self::sync, but without releasing the write lock in between.

Errors

An error is returned if writing to disk fails due to an I/O error.

Panics

This method panics if syncing fails irrecoverably.

pub fn commits(&self) -> impl Iterator<Item = Result<StoredCommit, Traversal>>

Obtain an iterator which traverses the log from the start, yielding StoredCommits.

The returned iterator is not aware of segment rotation. That is, if a new segment is created after this method returns, the iterator will not traverse it.

Commits appended to the log while it is being traversed are generally visible to the iterator. Upon encountering io::ErrorKind::UnexpectedEof, however, a new iterator should be created using Self::commits_from with the last transaction offset yielded.

Note that the very last StoredCommit in a commitlog may be corrupt (e.g. due to a partial write to disk), but a subsequent append will bring the log into a consistent state.

This means that, when this iterator yields an Err value, the consumer may want to check if the iterator is exhausted (by calling next()) before treating the Err value as an application error.

pub fn commits_from( &self, offset: u64, ) -> impl Iterator<Item = Result<StoredCommit, Traversal>>

Obtain an iterator starting from transaction offset offset, yielding StoredCommits.

Similar to Self::commits but will skip until the offset is contained in the next StoredCommit to yield.

Note that the first StoredCommit yielded is the first commit containing the given transaction offset, i.e. its min_tx_offset may be smaller than offset.

pub fn existing_segment_offsets(&self) -> Result<Vec<u64>>

Get a list of segment offsets, sorted in ascending order.

pub fn compress_segments(&self, offsets: &[u64]) -> Result<()>

Compress the segments at the offsets provided, marking them as immutable.

pub fn reset(self) -> Result<Self>

Remove all data from the log and reopen it.

Log segments are deleted starting from the newest. As multiple segments cannot be deleted atomically, the log may not be completely empty if the method returns an error.

Note that the method consumes self to ensure the log is not modified while resetting.

pub fn reset_to(self, offset: u64) -> Result<Self>

Remove all data past the given transaction offset from the log and reopen it.

Like with Self::reset, it may happen that not all segments newer than offset can be deleted.

If the method returns successfully, the most recent Commit in the log will contain the transaction at offset.

Note that the method consumes self to ensure the log is not modified while resetting.

pub fn size_on_disk(&self) -> Result<u64>

Determine the size on disk of this commitlog.

impl<T: Encode> Commitlog<T>

pub fn append(&self, txdata: T) -> Result<(), T>

Append the record txdata to the log.

If the internal buffer exceeds Options::max_records_in_commit, the argument is returned in an Err. The caller should Self::flush the log and try again.

In case the log is appended to from multiple threads, this may result in a busy loop trying to acquire a slot in the buffer. In such scenarios, Self::append_maybe_flush is preferable.

pub fn append_maybe_flush(&self, txdata: T) -> Result<(), Append<T>>

Append the record txdata to the log.

The txdata payload is buffered in memory until either:

• Self::flush is called explicitly, or
• Options::max_records_in_commit is exceeded

In the latter case, Self::append flushes implicitly, before appending the txdata argument.

I.e. the argument is not guaranteed to be flushed after the method returns. If that is desired, Self::flush must be called explicitly.

Errors

If the log needs to be flushed, but an I/O error occurs, ownership of txdata is returned back to the caller alongside the io::Error.

The value can then be used to retry appending.

pub fn transactions<'a, D>( &self, de: &'a D, ) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a

where D: Decoder<Record = T>, D::Error: From<Traversal>, T: 'a,

Obtain an iterator which traverses the log from the start, yielding Transactions.

The provided decoder’s Decoder::decode_record method will be called Commit::n times per Commit to obtain the individual transaction payloads.

Like Self::commits, the iterator is not aware of segment rotation. That is, if a new segment is created after this method returns, the iterator will not traverse it.

Transactions appended to the log while it is being traversed are generally visible to the iterator. Upon encountering io::ErrorKind::UnexpectedEof, however, a new iterator should be created using Self::transactions_from with the last transaction offset yielded.

Note that the very last Commit in a commitlog may be corrupt (e.g. due to a partial write to disk), but a subsequent append will bring the log into a consistent state.

This means that, when this iterator yields an Err value, the consumer may want to check if the iterator is exhausted (by calling next()) before treating the Err value as an application error.

pub fn transactions_from<'a, D>( &self, offset: u64, de: &'a D, ) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a

where D: Decoder<Record = T>, D::Error: From<Traversal>, T: 'a,

Obtain an iterator starting from transaction offset offset, yielding Transactions.

Similar to Self::transactions but will skip until the provided offset, i.e. the first Transaction yielded will be the transaction with offset offset.

pub fn fold_transactions<D>(&self, de: D) -> Result<(), D::Error>

where D: Decoder, D::Error: From<Traversal>,

Traverse the log from the start and “fold” its transactions into the provided Decoder.

A Decoder is a stateful object due to the requirement to store schema information in the log itself. That is, a Decoder may need to be able to resolve transaction schema information dynamically while traversing the log.

This is equivalent to “replaying” a log into a database state. In this scenario, it is not interesting to consume the Transaction payload as an iterator.

This method allows the use of a Decoder which returns zero-sized data (e.g. Decoder<Record = ()>), as it will not allocate the commit payload into a struct.

Note that, unlike Self::transactions, this method will ignore a corrupt commit at the very end of the traversed log.

pub fn fold_transactions_from<D>( &self, offset: u64, de: D, ) -> Result<(), D::Error>

where D: Decoder, D::Error: From<Traversal>,

Traverse the log from the given transaction offset and “fold” its transactions into the provided Decoder.

Similar to Self::fold_transactions but will skip until the provided offset, i.e. the first tx_offset passed to Decoder::decode_record will be equal to offset.