Struct spacetimedb_commitlog::Commitlog
source · pub struct Commitlog<T> { /* private fields */ }
Expand description
The canonical commitlog, backed by on-disk log files.
Records in the log are of type T
, which canonically is instantiated to
[Txdata
].
Implementations§
source§impl<T> Commitlog<T>
impl<T> Commitlog<T>
sourcepub fn open(root: impl Into<PathBuf>, opts: Options) -> Result<Self>
pub fn open(root: impl Into<PathBuf>, opts: Options) -> Result<Self>
Open the log at root directory root
with Options
.
The root directory must already exist.
Note that opening a commitlog involves I/O: some consistency checks are performed, and the next writing position is determined.
This is only necessary when opening the commitlog for writing. See the free-standing functions in this module for how to traverse a read-only commitlog.
sourcepub fn max_committed_offset(&self) -> Option<u64>
pub fn max_committed_offset(&self) -> Option<u64>
Determine the maximum transaction offset considered durable.
The offset is None
if the log hasn’t been flushed to disk yet.
sourcepub fn sync(&self) -> Result<Option<u64>>
pub fn sync(&self) -> Result<Option<u64>>
Sync all OS-buffered writes to disk.
Note that this does not write outstanding records to disk.
Use Self::flush_and_sync
or call Self::flush
prior to this
method to ensure all data is on disk.
Returns the maximum transaction offset which is considered durable after
this method returns successfully. The offset is None
if the log hasn’t
been flushed to disk yet.
sourcepub fn flush(&self) -> Result<Option<u64>>
pub fn flush(&self) -> Result<Option<u64>>
Write all outstanding transaction records to disk.
Note that this does not force the OS to sync the data to disk.
Use Self::flush_and_sync
or call Self::sync
after this method
to ensure all data is on disk.
Returns the maximum transaction offset written to disk. The offset is
None
if the log is empty and no data was pending to be flushed.
Repeatedly calling this method may return the same value.
sourcepub fn flush_and_sync(&self) -> Result<Option<u64>>
pub fn flush_and_sync(&self) -> Result<Option<u64>>
Write all outstanding transaction records to disk and flush OS buffers.
Equivalent to calling Self::flush
followed by Self::sync
, but
without releasing the write lock in between.
sourcepub fn commits(&self) -> impl Iterator<Item = Result<Commit, Traversal>>
pub fn commits(&self) -> impl Iterator<Item = Result<Commit, Traversal>>
Obtain an iterator which traverses the log from the start, yielding
Commit
s.
The returned iterator is not aware of segment rotation. That is, if a new segment is created after this method returns, the iterator will not traverse it.
Commits appended to the log while it is being traversed are generally
visible to the iterator. Upon encountering io::ErrorKind::UnexpectedEof
,
however, a new iterator should be created using Self::commits_from
with the last transaction offset yielded.
Note that the very last Commit
in a commitlog may be corrupt (e.g.
due to a partial write to disk), but a subsequent append
will bring
the log into a consistent state.
This means that, when this iterator yields an Err
value, the consumer
may want to check if the iterator is exhausted (by calling next()
)
before treating the Err
value as an application error.
sourcepub fn commits_from(
&self,
offset: u64
) -> impl Iterator<Item = Result<Commit, Traversal>>
pub fn commits_from( &self, offset: u64 ) -> impl Iterator<Item = Result<Commit, Traversal>>
Obtain an iterator starting from transaction offset offset
, yielding
Commit
s.
Similar to Self::commits
but will skip until the offset is contained
in the next Commit
to yield.
Note that the first Commit
yielded is the first commit containing
the given transaction offset, i.e. its min_tx_offset
may be smaller
than offset
.
sourcepub fn reset(self) -> Result<Self>
pub fn reset(self) -> Result<Self>
Remove all data from the log and reopen it.
Log segments are deleted starting from the newest. As multiple segments cannot be deleted atomically, the log may not be completely empty if the method returns an error.
Note that the method consumes self
to ensure the log is not modified
while resetting.
sourcepub fn reset_to(self, offset: u64) -> Result<Self>
pub fn reset_to(self, offset: u64) -> Result<Self>
Remove all data past the given transaction offset
from the log and
reopen it.
Like with Self::reset
, it may happen that not all segments newer
than offset
can be deleted.
If the method returns successfully, the most recent Commit
in the
log will contain the transaction at offset
.
Note that the method consumes self
to ensure the log is not modified
while resetting.
sourcepub fn size_on_disk(&self) -> Result<u64>
pub fn size_on_disk(&self) -> Result<u64>
Determine the size on disk of this commitlog.
source§impl<T: Encode> Commitlog<T>
impl<T: Encode> Commitlog<T>
sourcepub fn append(&self, txdata: T) -> Result<(), T>
pub fn append(&self, txdata: T) -> Result<(), T>
Append the record txdata
to the log.
If the internal buffer exceeds Options::max_records_in_commit
, the
argument is returned in an Err
. The caller should Self::flush
the
log and try again.
In case the log is appended to from multiple threads, this may result in
a busy loop trying to acquire a slot in the buffer. In such scenarios,
Self::append_maybe_flush
is preferable.
sourcepub fn append_maybe_flush(&self, txdata: T) -> Result<(), Append<T>>
pub fn append_maybe_flush(&self, txdata: T) -> Result<(), Append<T>>
Append the record txdata
to the log.
The txdata
payload is buffered in memory until either:
Self::flush
is called explicitly, orOptions::max_records_in_commit
is exceeded
In the latter case, Self::append
flushes implicitly, before
appending the txdata
argument.
I.e. the argument is not guaranteed to be flushed after the method
returns. If that is desired, Self::flush
must be called explicitly.
§Errors
If the log needs to be flushed, but an I/O error occurs, ownership of
txdata
is returned back to the caller alongside the io::Error
.
The value can then be used to retry appending.
sourcepub fn transactions<'a, D>(
&self,
de: &'a D
) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a
pub fn transactions<'a, D>( &self, de: &'a D ) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a
Obtain an iterator which traverses the log from the start, yielding
Transaction
s.
The provided decoder
’s Decoder::decode_record
method will be
called Commit::n
times per Commit
to obtain the individual
transaction payloads.
Like Self::commits
, the iterator is not aware of segment rotation.
That is, if a new segment is created after this method returns, the
iterator will not traverse it.
Transactions appended to the log while it is being traversed are
generally visible to the iterator. Upon encountering io::ErrorKind::UnexpectedEof
,
however, a new iterator should be created using Self::transactions_from
with the last transaction offset yielded.
Note that the very last Commit
in a commitlog may be corrupt (e.g.
due to a partial write to disk), but a subsequent append
will bring
the log into a consistent state.
This means that, when this iterator yields an Err
value, the consumer
may want to check if the iterator is exhausted (by calling next()
)
before treating the Err
value as an application error.
sourcepub fn transactions_from<'a, D>(
&self,
offset: u64,
de: &'a D
) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a
pub fn transactions_from<'a, D>( &self, offset: u64, de: &'a D ) -> impl Iterator<Item = Result<Transaction<T>, D::Error>> + 'a
Obtain an iterator starting from transaction offset offset
, yielding
Transaction
s.
Similar to Self::transactions
but will skip until the provided
offset
, i.e. the first Transaction
yielded will be the transaction
with offset offset
.
sourcepub fn fold_transactions<D>(&self, de: D) -> Result<(), D::Error>
pub fn fold_transactions<D>(&self, de: D) -> Result<(), D::Error>
Traverse the log from the start and “fold” its transactions into the
provided Decoder
.
A Decoder
is a stateful object due to the requirement to store
schema information in the log itself. That is, a Decoder
may need to
be able to resolve transaction schema information dynamically while
traversing the log.
This is equivalent to “replaying” a log into a database state. In this
scenario, it is not interesting to consume the Transaction
payload
as an iterator.
This method allows the use of a Decoder
which returns zero-sized
data (e.g. Decoder<Record = ()>
), as it will not allocate the commit
payload into a struct.
Note that, unlike [Self::transaction
], this method will ignore a
corrupt commit at the very end of the traversed log.
sourcepub fn fold_transactions_from<D>(
&self,
offset: u64,
de: D
) -> Result<(), D::Error>
pub fn fold_transactions_from<D>( &self, offset: u64, de: D ) -> Result<(), D::Error>
Traverse the log from the given transaction offset and “fold” its
transactions into the provided Decoder
.
Similar to Self::fold_transactions
but will skip until the provided
offset
, i.e. the first tx_offset
passed to Decoder::decode_record
will be equal to offset
.
source§impl<T: Send + Sync + 'static> Commitlog<T>
impl<T: Send + Sync + 'static> Commitlog<T>
sourcepub fn flush_and_sync_every(
self: Arc<Self>,
period: Duration
) -> Receiver<Result<Option<u64>>>
pub fn flush_and_sync_every( self: Arc<Self>, period: Duration ) -> Receiver<Result<Option<u64>>>
Call Self::flush_and_sync
periodically.
Returns a watch::Receiver
yielding the maximum durable transaction
offset after each invocation of Self::flush_and_sync
. The item type
is a Result
, so as to allow the caller to be notified of I/O errors.
The interval loop terminates when all receivers have been dropped. Note that this does not happen promptly.
Auto Trait Implementations§
impl<T> !Freeze for Commitlog<T>
impl<T> RefUnwindSafe for Commitlog<T>
impl<T> Send for Commitlog<T>where
T: Send,
impl<T> Sync for Commitlog<T>
impl<T> Unpin for Commitlog<T>where
T: Unpin,
impl<T> UnwindSafe for Commitlog<T>
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
source§impl<T> IntoEither for T
impl<T> IntoEither for T
source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moresource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read more