Struct ExtractDb 

pub struct ExtractDb<V>
where
    V: Eq + Hash + Clone + 'static + Send + Sync + Encode + for<'a> Decode<'a>,
{ /* private fields */ }

ExtractDb is a thread-safe, in-memory hash store supporting concurrent fetches and writes.

ExtractDb only supplies a push & fetch interface where both are &self. Once data is inserted it can never be removed. Persistence guaranteed.

You can think of it as a non-mutable concurrent VecDeque with unique values only.

Examples

Basic single threaded insertion example

use std::path::PathBuf;
use extractdb::ExtractDb;

let db: ExtractDb<i32> = ExtractDb::new(Some(PathBuf::from("/home/user/database_name")));

db.load_from_disk(true).unwrap();

db.push(100);

let item = db.fetch_next().unwrap();

db.save_to_disk().unwrap();

Autosaving

Multithreading capable, Arc<ExtractDb<V>> with background auto saving.

use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::time::Duration;
use extractdb::{CheckpointSettings, ExtractDb};

let db: Arc<ExtractDb<i32>> = Arc::new(ExtractDb::new(Some(PathBuf::from("/home/user/database_name"))));

db.load_from_disk(true).unwrap();

let shutdown_flag = Arc::new(AtomicBool::new(false));
let mut save_settings = CheckpointSettings::new(shutdown_flag.clone());
save_settings.minimum_changes = 10; // Minimum 10 changes
save_settings.check_delay = Duration::from_secs(5); // Check every 5 seconds

// Begin background saving thread
ExtractDb::background_checkpoints(save_settings, db.clone());

for i in 0..30 {
    db.push(i);
}

let item = db.fetch_next().unwrap();

// Exit background thread
shutdown_flag.store(true, Ordering::Relaxed);

Implementations

impl<V> ExtractDb<V>

where V: Eq + Hash + Clone + 'static + Send + Sync + Encode + for<'a> Decode<'a>,

pub fn new(database_directory: Option<PathBuf>) -> ExtractDb<V>

Creates a new ExtractDb

Arguments

database_directory: Allows saving of data to disk. This is optional!

Examples

use extractdb::ExtractDb;

// In-memory only example, set a path for save/loading.
let db: ExtractDb<String> = ExtractDb::new(None);

assert_eq!(db.push("Hello ExtractDb!".to_string()), true);

pub fn new_with_shards( shard_count: usize, database_directory: Option<PathBuf>, ) -> ExtractDb<V>

Creates a new ExtractDb with a specific internal sharding amount

Arguments

shard_count: Shards to be used internally. Think more shards = more concurrency, vice versa.

database_directory: Allows saving of data to disk. This is optional!

Examples

use extractdb::ExtractDb;

let db: ExtractDb<String> = ExtractDb::new_with_shards(32, None);

assert_eq!(db.push("Hello ExtractDb with custom shards!".to_string()), true);

pub fn push(&self, value: V) -> bool

Pushes data V into the internal sharded hashset.

Returns

True: if data has successfully inserted into a hashset

False: if data has already been added to a hashset, or if the internal shard is poisoned

Examples

use extractdb::ExtractDb;

let db: ExtractDb<i32> = ExtractDb::new(None);

assert_eq!(db.push(100), true);
assert_eq!(db.push(100), false);
assert_eq!(db.internal_count(), 1);

pub fn fetch_next(&self) -> Result<&V, Box<dyn Error + '_>>

Fetches a unique item from an internal queue

This function may act as a FIFO during low contention scenarios. Order is not guaranteed.

Returns

V A reference of the internal item

Errors

Box<dyn Error + '_> may return if queue is empty or if loading has a critical error

Examples

use extractdb::ExtractDb;

let db: ExtractDb<String> = ExtractDb::new(None);

assert_eq!(db.push("hello world".to_string()), true);
assert_eq!(db.fetch_next().unwrap(), &"hello world".to_string());
assert_eq!(db.internal_count(), 1);
assert_eq!(db.fetch_count(), 0);

pub fn fetch_count(&self) -> usize

Get the current count of the fetch_next mutable queue

Returns

usize a total of all items loaded into the temporary fetch vector

Examples

use extractdb::ExtractDb;

let db: ExtractDb<u8> = ExtractDb::new(None);

assert_eq!(db.push(20), true);
assert_eq!(db.fetch_count(), 0); // No data is currently loaded
assert_eq!(db.fetch_next().unwrap(), &20); // Causes a load for the non-mutable vector
assert_ne!(db.fetch_count(), 1);

pub fn internal_count(&self) -> usize

Get the internal count of items in all shards. This represents the total amount of items in the database at any time.

This function is impacted by writes and may be slowed.

Returns

usize a total of all items in the entire sharded database.

Examples

use extractdb::ExtractDb;

let db: ExtractDb<u8> = ExtractDb::new(None);

for i in 0..128 {
    assert_eq!(db.push(i), true);
}
assert_eq!(db.internal_count(), 128);

pub fn save_to_disk(&self) -> Result<(), Box<dyn Error + Send + Sync>>

Saves all internal shard data into a serialized database directory.

This method of saving is based off a naive checkpoint based system. All data is overwritten during every save.

Errors

Box<dyn Error> may return if database directory is not set or if creating fails.

pub fn load_from_disk( &self, re_enqueue: bool, ) -> Result<(), Box<dyn Error + Send + Sync>>

Loads all shard-files back into internal memory

During a failure/corruption event all non-corrupted data will be loaded into memory before an error is omitted out.

Arguments

re_enqueue: Loads all data back into fetch queue.

Errors

Box<dyn Error + Send + Sync> may return if any form of corruption occurs, or if a shard size changes.

pub fn background_checkpoints( settings: CheckpointSettings, db: Arc<ExtractDb<V>>, )

Spawns a background thread to provide periodic save checkpoints onto disk. Only performs a save if a certain threshold of changes has occurred during the save interval. See CheckpointSettings

This function is meant for long-running applications/multithreaded instances.

Use check_delay within CheckpointSettings to determine the frequency in which change counts occur.

Use minimum_changes within CheckpointSettings to determine the minimum amount of items inserted before a safe event occurs.

Use shutdown_flag within CheckpointSettings to remotely shut down this thread in a safe manner. False = Running, True = Please stop

Different behavior

You do not necessarily need to use this function for auto-saving. All methods used are publicly available and easily re-implementable. See source.

Parameters

settings: CheckpointSettings determines the check rate and minimum change for disk saving.

db: Arc<ExtractDb<V>> instance of ExtractDb in a shared instance

Examples

use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use extractdb::{CheckpointSettings, ExtractDb};

let db: Arc<ExtractDb<u8>> = Arc::new(ExtractDb::new(None));

let shutdown_flag = Arc::new(AtomicBool::new(false));
let mut save_settings = CheckpointSettings::new(shutdown_flag.clone());
save_settings.minimum_changes = 1000;

// Will now check for 1000 minimum changes every 30seconds (default).
ExtractDb::background_checkpoints(save_settings, db.clone());

db.push(127);

// Gracefully shutdown a background thread
shutdown_flag.store(true, Ordering::Relaxed);

Trait Implementations

impl<V> Default for ExtractDb<V>

where V: Eq + Hash + Clone + 'static + Send + Sync + Encode + for<'a> Decode<'a>,

fn default() -> Self

Returns the “default value” for a type.