Struct indexkv::Store

pub struct Store<K, T> where
    K: Key,
    T: Serialize + DeserializeOwned,  { /* private fields */ }

Implementations

impl<K, T> Store<K, T> where
    K: Key,
    T: Serialize + DeserializeOwned + Debug, 

pub async fn new(path: impl AsRef<Path>) -> Result<Self, Error<K>>

Allocates a new Store containing no values. Opens a file descriptor to path.

pub async fn is_valid(&mut self) -> Result<bool, Error<K>>

Reads the header and checks to see whether or not the store is valid; at this time, this means making sure the format is fully up-to-date and writing completed.

pub async fn stream_values(
    &self
) -> Result<impl Stream<Item = Result<T, Error<K>>>, Error<K>>

Provides a sequential stream of all values in the Store. In relational database parlance, this would be a tablescan.

pub async fn get(&mut self, key: K) -> Result<T, Error<K>>

Looks up a single value by key. Will attempt to cache data positions in the backing file to minimize disk seeking. Returns Error::NotFound when key does not exist in the Store.

pub async fn get_many(
    &mut self,
    keys_in: &[K]
) -> Result<HashMap<K, T>, Error<K>>

Looks up multiple values by key. Will find (uncached) keys in sorted order to minimize disk seeking back and forth. Does not return an error when key does not exist; calling code should check for Some vs None in the resulting HashMap for fallibility.

pub async fn write<S>(
    &mut self,
    input_stream: S,
    index_size_hint: usize
) -> Result<(), StreamError<S::Error, K>> where
    S: TryStream<Ok = (K, T)> + Unpin,
    S::Error: Error, 

Consumes a TryStream of (key, value) pairs. Sequentially writes all the values to disk, noting their positions, then goes back and sequentially writes down the sorted index.

Making effective use of this error handling mechanism is complex; an example is included below.

The Error type is usually inferrable. A recommended pattern is to have it be your crate’s Error type, or anyhow::Error.

index_size_hint is used to preallocate the in-memory index that is eventually written to disk, which is strictly a performance optimization. If it’s unknown, passing a 0 is fine.

Example:

use std::io;
use futures::TryStreamExt;
use indexkv::{Store, StreamError};

#[derive(Debug, thiserror::Error)]
enum MyError {
	#[error("I/O error")]
	IO(#[from] io::Error),
	#[error("cache error")]
	Cache(#[from] indexkv::Error<u64>)
}

impl From<StreamError<MyError, u64>> for MyError {
	fn from(other: StreamError<MyError, u64>) -> Self {
		match other {
			StreamError::Caller(e) => e,
			StreamError::Internal(e) => MyError::from(e)
		}
	}
}

async fn write() -> Result<(), MyError> {
	let path = tempfile::NamedTempFile::new().unwrap().into_temp_path();
	let mut store: Store<u64, String> = Store::new(path).await.unwrap();

	let stream = futures::stream::iter(
		vec![Ok("zero"), Ok("one"), Err(io::Error::from(io::ErrorKind::BrokenPipe)), Ok("two")]
			.into_iter()
			.enumerate()
			.map(|(i, r)| match r {
				Ok(v) => Ok((i as u64, v.to_string())),
				Err(e) => Err(e)
			})
	).err_into::<MyError>();
	store.write(stream, 0).await?;
	Ok(())
}

#[tokio::main]
async fn main() {
	let result = write().await;
	assert!(matches!(
		result,
		Err(MyError::IO(_))
	));
}

pub async fn write_infallible<S>(
    &mut self,
    input_stream: S,
    index_size_hint: usize
) -> Result<(), Error<K>> where
    S: Stream<Item = (K, T)> + Unpin, 

For cases where the incoming data stream is infallible, this allows you to avoid that portion of error handling.