Struct Log

pub struct Log {
    pub dir: GenericPath,
    /* private fields */
}

An append-only storage with indexes and integrity checks.

The Log is backed by a directory in the filesystem. The directory includes:

• An append-only “log” file. It can be seen as a serialization result of an append-only list of byte slices. Each byte slice has a checksum.
• Multiple user-defined indexes. Each index has an append-only on-disk radix-tree representation and a small, separate, non-append-only checksum file.
• A small “metadata” file which records the logic lengths (in bytes) for the log and index files.

Reading is lock-free because the log and indexes are append-only. Writes are buffered in memory. Flushing in-memory parts to disk requires taking a flock on the directory.

Fields

dir: GenericPath

Implementations

impl Log

pub fn open<P: AsRef<Path>>(dir: P, index_defs: Vec<IndexDef>) -> Result<Self>

Construct Log at given directory. Incrementally build up specified indexes.

Create the Log if it does not exist.

See OpenOptions::open for details.

pub fn append<T: AsRef<[u8]>>(&mut self, data: T) -> Result<()>

Append an entry in-memory. Update related indexes in-memory.

The memory part is not shared. Therefore other Log instances won’t see the change immediately.

To write in-memory entries and indexes to disk, call Log::sync.

pub fn clear_dirty(&mut self) -> Result<()>

Remove dirty (in-memory) state. Restore the Log to the state as if it’s just loaded from disk without modifications.

pub fn try_clone(&self) -> Result<Self>

Return a cloned Log with pending in-memory changes.

pub fn try_clone_without_dirty(&self) -> Result<Self>

Return a cloned Log without pending in-memory changes.

This is logically equivalent to calling clear_dirty immediately on the result after try_clone, but potentially cheaper.

pub fn sync(&mut self) -> Result<u64>

Load the latest data from disk. Write in-memory entries to disk.

After writing, update on-disk indexes. These happen in a same critical section, protected by a lock on the directory.

Even if Log::append is never called, this function has a side effect updating the Log to contain latest entries on disk.

Other Log instances living in a same process or other processes won’t be notified about the change and they can only access the data “snapshotted” at open time.

Return the size of the updated primary log file in bytes.

For in-memory-only Logs, this function does nothing, and returns 0.

pub fn is_changed_on_disk(&self) -> bool

Returns true if sync will load more data on disk.

This function is optimized to be called frequently. It does not access the filesystem directly, but communicate using a shared mmap buffer.

This is not about testing buffered pending changes. To access buffered pending changes, use Log::iter_dirty instead.

For an in-memory Log, this always returns false.

pub fn flush(&mut self) -> Result<u64>

Renamed. Use Log::sync instead.

pub fn slice_to_bytes(&self, slice: &[u8]) -> Bytes

Convert a slice to Bytes. Do not copy the slice if it’s from the main on-disk buffer.

pub fn index_slice_to_bytes(&self, index_id: usize, slice: &[u8]) -> Bytes

Convert a slice to Bytes. Do not copy the slice if it’s from the specified index buffer.

pub fn rebuild_indexes(self, force: bool) -> Result<String>

Rebuild indexes.

If force is false, then indexes that pass the checksum check will not be rebuilt. Otherwise, they will be rebuilt regardless.

Setting force to true might reduce the size used by the index files. But that is more expensive.

The function consumes the Log object, since it is hard to recover from an error case.

Return message useful for human consumption.

pub fn lookup<K: AsRef<[u8]>>( &self, index_id: usize, key: K, ) -> Result<LogLookupIter<'_>>

Look up an entry using the given index. The index_id is the index of index_defs passed to Log::open.

Return an iterator of Result<&[u8]>, in reverse insertion order.

pub fn lookup_prefix<K: AsRef<[u8]>>( &self, index_id: usize, prefix: K, ) -> Result<LogRangeIter<'_>>

Look up keys and entries using the given prefix. The index_id is the index of index_defs passed to Log::open.

Return an iterator that yields (key, iter), where key is the full key, iter is LogLookupIter that allows iteration through matched entries.

pub fn lookup_range<'a>( &self, index_id: usize, range: impl RangeBounds<&'a [u8]>, ) -> Result<LogRangeIter<'_>>

Look up keys and entries by querying a specified index about a specified range.

The index_id is the index of index_defs defined by OpenOptions.

Return an iterator that yields (key, iter), where key is the full key, iter is LogLookupIter that allows iteration through entries matching that key.

pub fn lookup_prefix_hex<K: AsRef<[u8]>>( &self, index_id: usize, hex_prefix: K, ) -> Result<LogRangeIter<'_>>

Look up keys and entries using the given hex prefix. The length of the hex string can be odd.

Return an iterator that yields (key, iter), where key is the full key, iter is LogLookupIter that allows iteration through matched entries.

pub fn iter(&self) -> LogIter<'_>

Return an iterator for all entries.

pub fn iter_dirty(&self) -> LogIter<'_>

Return an iterator for in-memory entries that haven’t been flushed to disk.

For in-memory Logs, this is the same as Log::iter.

pub fn index_func<'a>( &self, index_id: usize, entry: &'a [u8], ) -> Result<Vec<Cow<'a, [u8]>>>

Applies the given index function to the entry data and returns the index keys.

pub fn fold(&self, fold_id: usize) -> Result<&dyn Fold>

Return the fold state after calling accumulate on all (on-disk and in-memory) entries in insertion order.

The fold function is the fold_id-th (0-based) FoldDef in OpenOptions.

pub fn path(&self) -> &GenericPath

Return the reference to the GenericPath used to crate the Log.

pub fn version(&self) -> (u64, u64)

Return the version in (epoch, length) form.

The version is maintained exclusively by indexedlog and cannot be changed directly via public APIs. Appending data bumps length. Rewriting data changes epoch.

See also crate::multi::MultiLog::version.

Trait Implementations

impl Debug for Log

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.