Struct pearl::Storage

pub struct Storage<K>where
    for<'a> K: Key<'a>,{ /* private fields */ }

A main storage struct.

This type is clonable, cloning it will only create a new reference, not a new storage. Storage has a type kindeter K. To perform read/write operations K must implement Key trait.

Examples

use pearl::{Storage, Builder, Key, ArrayKey};

#[tokio::main]
async fn main() {
    let mut storage: Storage<ArrayKey<8>> = Builder::new()
        .work_dir("/tmp/pearl/")
        .max_blob_size(1_000_000)
        .max_data_in_blob(1_000_000_000)
        .blob_file_name_prefix("pearl-test")
        .build()
        .unwrap();
    storage.init().await.unwrap();
}

Implementations

impl<K> Storage<K>where
    for<'a> K: Key<'a> + 'static,

pub async fn init(&mut self) -> Result<()>

init() used to prepare all environment to further work.

Storage works in directory provided to builder. If directory don’t exist, storage creates it, otherwise tries to init existing storage.

Errors

Returns error in case of failures with IO operations or if some of the required kinds are missed.

pub async fn init_lazy(&mut self) -> Result<()>

init_lazy() used to prepare all environment to further work, but unlike init doesn’t set active blob, which means that first write may take time..

Storage works in directory provided to builder. If directory don’t exist, storage creates it, otherwise tries to init existing storage.

Errors

Returns error in case of failures with IO operations or if some of the required params are missed.

pub fn is_pending(&self) -> bool

Checks if there is a pending async operation Returns boolean value (true - if there is, false otherwise) Never falls

pub async fn has_active_blob(&self) -> bool

FIXME: maybe it would be better to add check of is_pending state of observer for all sync operations and return result in more appropriate way for that case (change Result on Result, where OpRes = Pending|Done|NotDone for example) Checks if active blob is set Returns boolean value Never falls

pub async fn try_create_active_blob(&self) -> Result<()>

Creates active blob NOTICE! This function works in current thread, so it may take time. To perform this asyncronously, use [create_active_blob_in_background()] Returns true if new blob was created else false

Errors

Fails if it’s not possible to create new blob [create_active_blob_in_background()]: struct.Storage.html#method.create_active_blob_async

pub async fn create_active_blob_in_background(&self)

Creates active blob NOTICE! This function returns immediately, so you can’t check result of operation. If you want be sure about operation’s result, use [try_create_active_blob()] [try_create_active_blob()]: struct.Storage.html#method.try_create_active_blob

pub async fn try_close_active_blob(&self) -> Result<()>

Dumps active blob NOTICE! This function works in current thread, so it may take time. To perform this asyncronously, use [close_active_blob_in_background()] Returns true if blob was really dumped else false

Errors

Fails if there are some errors during dump [close_active_blob_in_background()]: struct.Storage.html#method.create_active_blob_async

pub async fn close_active_blob_in_background(&self)

Dumps active blob NOTICE! This function returns immediately, so you can’t check result of operation. If you want be sure about operation’s result, use [try_close_active_blob()]

pub async fn try_restore_active_blob(&self) -> Result<()>

Sets last blob from closed blobs as active if there is no active blobs NOTICE! This function works in current thread, so it may take time. To perform this asyncronously, use [restore_active_blob_in_background()] Returns true if last blob was set as active as false

Errors

Fails if active blob is set or there is no closed blobs [restore_active_blob_in_background()]: struct.Storage.html#method.restore_active_blob_async

pub async fn restore_active_blob_in_background(&self)

Sets last blob from closed blobs as active if there is no active blobs NOTICE! This function returns immediately, so you can’t check result of operation. If you want be sure about operation’s result, use [try_restore_active_blob()] [try_restore_active_blob()]: struct.Storage.html#method.try_restore_active_blob

pub async fn write(&self, key: impl AsRef<K>, value: Vec<u8>) -> Result<()>

Writes data to active blob asyncronously. If active blob reaches it limit, creates new and closes old. NOTICE! First write into storage without active blob may take more time due to active blob creation

Examples

use pearl::{Builder, Storage, ArrayKey};

async fn write_data(storage: Storage<ArrayKey<8>>) {
    let key = ArrayKey::<8>::default();
    let data = b"async written to blob".to_vec();
    storage.write(key, data).await;
}

Errors

Fails with the same errors as write_with

pub async fn write_with(
    &self,
    key: impl AsRef<K>,
    value: Vec<u8>,
    meta: Meta
) -> Result<()>

Similar to write but with metadata

Examples

use pearl::{Builder, Meta, Storage, ArrayKey};

async fn write_data(storage: Storage<ArrayKey<8>>) {
    let key = ArrayKey::<8>::default();
    let data = b"async written to blob".to_vec();
    let mut meta = Meta::new();
    meta.insert("version".to_string(), b"1.0".to_vec());
    storage.write_with(&key, data, meta).await;
}

Errors

Fails if duplicates are not allowed and record already exists.

pub async fn free_excess_resources(&self) -> usize

Free all resources that may be freed without work interruption NOTICE! This function frees part of the resources in separate thread, so actual resources may be freed later

pub async fn inactive_index_memory(&self) -> usize

Get size in bytes of inactive indexes

pub async fn index_memory(&self) -> usize

Get size in bytes of all freeable resources

pub async fn read(&self, key: impl AsRef<K>) -> Result<Vec<u8>>

Reads the first found data matching given key.

Examples

use pearl::{Builder, Meta, Storage, ArrayKey};

async fn read_data(storage: Storage<ArrayKey<8>>) {
    let key = ArrayKey::<8>::default();
    let data = storage.read(key).await;
}

Errors

Same as read_with

pub async fn read_with(&self, key: impl AsRef<K>, meta: &Meta) -> Result<Vec<u8>>

Reads data matching given key and metadata

Examples

use pearl::{Builder, Meta, Storage, ArrayKey};

async fn read_data(storage: Storage<ArrayKey<8>>) {
    let key = ArrayKey::<8>::default();
    let mut meta = Meta::new();
    meta.insert("version".to_string(), b"1.0".to_vec());
    let data = storage.read_with(&key, &meta).await;
}

Errors

Return error if record is not found.

pub async fn read_all(&self, key: impl AsRef<K>) -> Result<Vec<Entry>>

Returns entries with matching key

Errors

Fails after any disk IO errors.

pub async fn close(self) -> Result<()>

Stop blob updater and release lock file

Errors

Fails because of any IO errors

pub async fn blobs_count(&self) -> usize

blob_count returns exact number of closed blobs plus one active, if there is some. It locks on inner structure, so it much slower than next_blob_id.

Examples

use pearl::{Builder, Storage, ArrayKey};

async fn check_blobs_count(storage: Storage<ArrayKey<8>>) {
    assert_eq!(storage.blobs_count().await, 1);
}

pub async fn active_index_memory(&self) -> usize

active_index_memory returns the amount of memory used by blob to store active indices

pub async fn disk_used(&self) -> u64

disk_used returns amount of disk space occupied by storage related files

pub fn next_blob_id(&self) -> usize

Returns next blob ID. If pearl dir structure wasn’t changed from the outside, returned number is equal to blobs_count. But this method doesn’t require lock. So it is much faster than blobs_count.

pub async fn contains(&self, key: impl AsRef<K>) -> Result<bool>

contains is used to check whether a key is in storage. Slower than check_bloom, because doesn’t prevent disk IO operations. contains returns either “definitely in storage” or “definitely not”.

Errors

Fails because of any IO errors

pub async fn check_filters(&self, key: impl AsRef<K>) -> Option<bool>

check_filters is used to check whether a key is in storage. Range (min-max test) and bloom filters are used. If bloom filter opt out and range filter passes, returns None. False positive results are possible, but false negatives are not. In other words, check_filters returns either “possibly in storage” or “definitely not”.

pub async fn records_count(&self) -> usize

Total records count in storage.

pub async fn records_count_detailed(&self) -> Vec<(usize, usize)>

Records count per blob. Format: (blob_id, count). Last value is from active blob.

pub async fn records_count_in_active_blob(&self) -> Option<usize>

Records count in active blob. Returns None if active blob not set or any IO error occured.

pub async fn fsyncdata(&self) -> IOResult<()>

Syncronizes data and metadata of the active blob with the filesystem. Like tokio::std::fs::File::sync_data, this function will attempt to ensure that in-core data reaches the filesystem before returning. May not syncronize file metadata to the file system.

Errors

Fails because of any IO errors.

pub async fn force_update_active_blob(
    &self,
    predicate: fn(_: Option<ActiveBlobStat>) -> bool
)

Force updates active blob on new one to dump index of old one on disk and free RAM. This function was used previously instead of [close_active_blob_in_background()] Creates new active blob.

Errors

Fails because of any IO errors. Or if there are some problems with syncronization. [close_active_blob_in_background()]: struct.Storage.html#method.close_active_blob_async

pub async fn mark_all_as_deleted(&self, key: impl AsRef<K>) -> Result<u64>

Mark as deleted entries with matching key

Errors

Fails after any disk IO errors.

Trait Implementations

impl<K> BloomProvider<K> for Storage<K>where
    for<'a> K: Key<'a> + 'static,

type Filter = <Blob<K> as BloomProvider<K>>::Filter

Inner filter type

fn check_filter<'life0, 'life1, 'async_trait>(
    &'life0 self,
    item: &'life1 K
) -> Pin<Box<dyn Future<Output = FilterResult> + Send + 'async_trait>>where
    'life0: 'async_trait,
    'life1: 'async_trait,
    Self: 'async_trait,

Check if element in filter

fn check_filter_fast(&self, _item: &K) -> FilterResult

Check if element in filter

fn offload_buffer<'life0, 'async_trait>(
    &'life0 mut self,
    needed_memory: usize,
    level: usize
) -> Pin<Box<dyn Future<Output = usize> + Send + 'async_trait>>where
    'life0: 'async_trait,
    Self: 'async_trait,

Returns freed memory

fn get_filter<'life0, 'async_trait>(
    &'life0 self
) -> Pin<Box<dyn Future<Output = Option<Self::Filter>> + Send + 'async_trait>>where
    'life0: 'async_trait,
    Self: 'async_trait,

Returns overall filter

fn get_filter_fast(&self) -> Option<&Self::Filter>

Returns overall filter

fn filter_memory_allocated<'life0, 'async_trait>(
    &'life0 self
) -> Pin<Box<dyn Future<Output = usize> + Send + 'async_trait>>where
    'life0: 'async_trait,
    Self: 'async_trait,

Returns allocated memory

impl<K: Clone> Clone for Storage<K>where
    for<'a> K: Key<'a>,

fn clone(&self) -> Storage<K>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<K: Debug> Debug for Storage<K>where
    for<'a> K: Key<'a>,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.