Struct pearl::Storage

source ·
pub struct Storage<K>
where for<'a> K: Key<'a>,
{ /* private fields */ }
Expand description

A main storage struct.

Storage has a generic parameter K representing the type of Key. To perform read/write operations K must implement Key trait.

Examples

use pearl::{Storage, Builder, Key, ArrayKey};

#[tokio::main]
async fn main() {
    let mut storage: Storage<ArrayKey<8>> = Builder::new()
        .work_dir("/tmp/pearl/")
        .max_blob_size(1_000_000)
        .max_data_in_blob(1_000_000_000)
        .blob_file_name_prefix("pearl-test")
        .build()
        .unwrap();
    storage.init().await.unwrap();
}

Implementations§

source§

impl<K> Storage<K>
where for<'a> K: Key<'a> + 'static,

source

pub async fn init(&mut self) -> Result<()>

init() used to prepare all environment to further work.

Storage works in directory provided to builder. If directory don’t exist, storage creates it, otherwise tries to init existing storage.

Errors

Returns error in case of failures with IO operations or if some of the required kinds are missed.

source

pub async fn init_lazy(&mut self) -> Result<()>

init_lazy() used to prepare all environment to further work, but unlike init doesn’t set active blob, which means that first write may take time..

Storage works in directory provided to builder. If directory don’t exist, storage creates it, otherwise tries to init existing storage.

Errors

Returns error in case of failures with IO operations or if some of the required params are missed.

source

pub async fn has_active_blob(&self) -> bool

FIXME: maybe it would be better to add check of is_pending state of observer for all sync operations and return result in more appropriate way for that case (change Result on Result, where OpRes = Pending|Done|NotDone for example) Checks if active blob is set Returns boolean value Never falls

source

pub async fn try_create_active_blob(&self) -> Result<()>

Creates active blob NOTICE! This function works in current thread, so it may take time. To perform this asyncronously, use [create_active_blob_in_background()] Returns true if new blob was created else false

Errors

Fails if it’s not possible to create new blob [create_active_blob_in_background()]: struct.Storage.html#method.create_active_blob_async

source

pub async fn create_active_blob_in_background(&self)

Creates active blob NOTICE! This function returns immediately, so you can’t check result of operation. If you want be sure about operation’s result, use [try_create_active_blob()] [try_create_active_blob()]: struct.Storage.html#method.try_create_active_blob

source

pub async fn try_close_active_blob(&self) -> Result<()>

Dumps active blob NOTICE! This function works in current thread, so it may take time. To perform this asyncronously, use [close_active_blob_in_background()] Returns true if blob was really dumped else false

Errors

Fails if there are some errors during dump [close_active_blob_in_background()]: struct.Storage.html#method.create_active_blob_async

source

pub async fn close_active_blob_in_background(&self)

Dumps active blob NOTICE! This function returns immediately, so you can’t check result of operation. If you want be sure about operation’s result, use [try_close_active_blob()]

source

pub async fn try_restore_active_blob(&self) -> Result<()>

Sets last blob from closed blobs as active if there is no active blobs NOTICE! This function works in current thread, so it may take time. To perform this asyncronously, use [restore_active_blob_in_background()] Returns true if last blob was set as active as false

Errors

Fails if active blob is set or there is no closed blobs [restore_active_blob_in_background()]: struct.Storage.html#method.restore_active_blob_async

source

pub async fn restore_active_blob_in_background(&self)

Sets last blob from closed blobs as active if there is no active blobs NOTICE! This function returns immediately, so you can’t check result of operation. If you want be sure about operation’s result, use [try_restore_active_blob()] [try_restore_active_blob()]: struct.Storage.html#method.try_restore_active_blob

source

pub async fn write( &self, key: impl AsRef<K>, value: Bytes, timestamp: BlobRecordTimestamp ) -> Result<()>

Writes data to active blob asyncronously. If active blob reaches it limit, creates new and closes old. NOTICE! First write into storage without active blob may take more time due to active blob creation

Examples
use pearl::{Builder, Storage, ArrayKey, BlobRecordTimestamp};

async fn write_data(storage: Storage<ArrayKey<8>>) {
    let key = ArrayKey::<8>::default();
    let data = b"async written to blob".to_vec().into();
    let timestamp = BlobRecordTimestamp::now();
    storage.write(key, data, timestamp).await;
}
Errors

Fails with the same errors as write_with

source

pub async fn write_with( &self, key: impl AsRef<K>, value: Bytes, timestamp: BlobRecordTimestamp, meta: Meta ) -> Result<()>

Similar to write but with metadata

Examples
use pearl::{Builder, Meta, Storage, ArrayKey, BlobRecordTimestamp};

async fn write_data(storage: Storage<ArrayKey<8>>) {
    let key = ArrayKey::<8>::default();
    let data = b"async written to blob".to_vec().into();
    let timestamp = BlobRecordTimestamp::now();
    let mut meta = Meta::new();
    meta.insert("version".to_string(), b"1.0".to_vec());
    storage.write_with(&key, data, timestamp, meta).await;
}
Errors

Fails if duplicates are not allowed and record already exists.

source

pub async fn free_excess_resources(&self) -> usize

Free all resources that may be freed without work interruption NOTICE! This function frees part of the resources in separate thread, so actual resources may be freed later

source

pub async fn inactive_index_memory(&self) -> usize

Get size in bytes of inactive indexes

source

pub async fn index_memory(&self) -> usize

Get size in bytes of all freeable resources

source

pub async fn read(&self, key: impl AsRef<K>) -> Result<ReadResult<Bytes>>

Reads the first found data matching given key.

Examples
use pearl::{Builder, Meta, Storage, ArrayKey};

async fn read_data(storage: Storage<ArrayKey<8>>) {
    let key = ArrayKey::<8>::default();
    let data = storage.read(key).await;
}
Errors

Same as read_with

source

pub async fn read_with( &self, key: impl AsRef<K>, meta: &Meta ) -> Result<ReadResult<Bytes>>

Reads data matching given key and metadata

Examples
use pearl::{Builder, Meta, Storage, ArrayKey};

async fn read_data(storage: Storage<ArrayKey<8>>) {
    let key = ArrayKey::<8>::default();
    let mut meta = Meta::new();
    meta.insert("version".to_string(), b"1.0".to_vec());
    let data = storage.read_with(&key, &meta).await;
}
Errors

Return error if record is not found.

source

pub async fn read_all(&self, key: impl AsRef<K>) -> Result<Vec<Entry>>

Returns entries with matching key

Errors

Fails after any disk IO errors.

source

pub async fn read_all_with_deletion_marker( &self, key: impl AsRef<K> ) -> Result<Vec<Entry>>

Returns entries with matching key and deletion marker, if any

Errors

Fails after any disk IO errors.

source

pub async fn close(self) -> Result<()>

Stop blob updater and release lock file

Errors

Fails because of any IO errors

source

pub async fn blobs_count(&self) -> usize

blob_count returns exact number of closed blobs plus one active, if there is some. It locks on inner structure, so it much slower than next_blob_id.

Examples
use pearl::{Builder, Storage, ArrayKey};

async fn check_blobs_count(storage: Storage<ArrayKey<8>>) {
    assert_eq!(storage.blobs_count().await, 1);
}
source

pub fn corrupted_blobs_count(&self) -> usize

blob_count returns exact number of corrupted blobs.

source

pub async fn active_index_memory(&self) -> usize

active_index_memory returns the amount of memory used by blob to store active indices

source

pub async fn disk_used(&self) -> u64

disk_used returns amount of disk space occupied by storage related files

source

pub fn next_blob_id(&self) -> usize

Returns next blob ID. If pearl dir structure wasn’t changed from the outside, returned number is equal to blobs_count. But this method doesn’t require lock. So it is much faster than blobs_count.

source

pub async fn contains( &self, key: impl AsRef<K> ) -> Result<ReadResult<BlobRecordTimestamp>>

contains is used to check whether a key is in storage. Slower than check_bloom, because doesn’t prevent disk IO operations. contains returns either “definitely in storage” or “definitely not”. Also returns creation timestamp

Errors

Fails because of any IO errors

source

pub async fn check_filters(&self, key: impl AsRef<K>) -> Option<bool>

check_filters is used to check whether a key is in storage. Range (min-max test) and bloom filters are used. If bloom filter opt out and range filter passes, returns None. False positive results are possible, but false negatives are not. In other words, check_filters returns either “possibly in storage” or “definitely not”.

source

pub async fn records_count(&self) -> usize

Total records count in storage.

source

pub async fn records_count_detailed(&self) -> Vec<(usize, usize)>

Records count per blob. Format: (blob_id, count). Last value is from active blob.

source

pub async fn records_count_in_active_blob(&self) -> Option<usize>

Records count in active blob. Returns None if active blob not set or any IO error occured.

source

pub async fn fsyncdata(&self) -> IOResult<()>

Syncronizes data and metadata of the active blob with the filesystem. Like tokio::std::fs::File::sync_data, this function will attempt to ensure that in-core data reaches the filesystem before returning. May not syncronize file metadata to the file system.

Errors

Fails because of any IO errors.

source

pub async fn force_update_active_blob( &self, predicate: fn(_: Option<ActiveBlobStat>) -> bool )

Force updates active blob on new one to dump index of old one on disk and free RAM. This function was used previously instead of [close_active_blob_in_background()] Creates new active blob.

Errors

Fails because of any IO errors. Or if there are some problems with syncronization. [close_active_blob_in_background()]: struct.Storage.html#method.close_active_blob_async

source

pub async fn delete( &self, key: impl AsRef<K>, timestamp: BlobRecordTimestamp, only_if_presented: bool ) -> Result<u64>

Delete entries with matching key

Errors

Fails after any disk IO errors.

source

pub async fn delete_with( &self, key: impl AsRef<K>, timestamp: BlobRecordTimestamp, meta: Meta, only_if_presented: bool ) -> Result<u64>

Delete entries with matching key. Appends metadata to deletion record

Errors

Fails after any disk IO errors.

Trait Implementations§

source§

impl<K> BloomProvider<K> for Storage<K>
where for<'a> K: Key<'a> + 'static,

§

type Filter = <Blob<K> as BloomProvider<K>>::Filter

Inner filter type
source§

fn check_filter<'life0, 'life1, 'async_trait>( &'life0 self, item: &'life1 K ) -> Pin<Box<dyn Future<Output = FilterResult> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Check if element in filter
source§

fn check_filter_fast(&self, _item: &K) -> FilterResult

Check if element in filter
source§

fn offload_buffer<'life0, 'async_trait>( &'life0 mut self, needed_memory: usize, level: usize ) -> Pin<Box<dyn Future<Output = usize> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Returns freed memory
source§

fn get_filter<'life0, 'async_trait>( &'life0 self ) -> Pin<Box<dyn Future<Output = Option<Self::Filter>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Returns overall filter
source§

fn get_filter_fast(&self) -> Option<&Self::Filter>

Returns overall filter
source§

fn filter_memory_allocated<'life0, 'async_trait>( &'life0 self ) -> Pin<Box<dyn Future<Output = usize> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait,

Returns allocated memory
source§

impl<K> Debug for Storage<K>
where for<'a> K: Key<'a> + Debug,

source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

Auto Trait Implementations§

§

impl<K> !RefUnwindSafe for Storage<K>

§

impl<K> Send for Storage<K>

§

impl<K> Sync for Storage<K>

§

impl<K> Unpin for Storage<K>

§

impl<K> !UnwindSafe for Storage<K>

Blanket Implementations§

source§

impl<T> Any for T
where T: 'static + ?Sized,

source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
source§

impl<T> Borrow<T> for T
where T: ?Sized,

source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
source§

impl<T> From<T> for T

source§

fn from(t: T) -> T

Returns the argument unchanged.

source§

impl<T, U> Into<U> for T
where U: From<T>,

source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

source§

impl<T> Same for T

§

type Output = T

Should always be Self
source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

§

type Error = Infallible

The type returned in the event of a conversion error.
source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.