Crate vec

Expand description

§BRK Vec

Size

A Vec (an array) that is stored on disk and thus which can be much larger than the available RAM.

Compared to a key/value store, the data stored is raw byte interpretation of the Vec’s values without any overhead which is very efficient. Additionally it uses close to no RAM when caching isn’t active and up to 100 MB when it is.

Compression is also available and built on top zstd to save even more space (from 0 to 75%). The tradeoff being slower reading speeds, especially random reading speeds. This is due to the data being stored in compressed pages of 16 KB, which means that if you to read even one value in that page you have to uncompress the whole page.

§Disclaimer

Portability will depend on the type of values.

Non bytes/slices types (u8, u16, …) will be read as slice in an unsafe manner (using std::slice::from_raw_parts) and thus have the endianness of the system. On the other hand, &[u8] should be inserted as is.

If portability is important to you, just create a wrapper struct which has custom get, push, … methods and does something like:

impl StorableVecU64 {
    pub fn push(&mut self, value: u64) {
        self.push(&value.to_be_bytes())
    }
}

§Example

use std::{fs, path::Path};

use brk_vec::{Compressed, StorableVec, Version};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let _ = fs::remove_dir_all("./vec");

    {
        let mut vec: StorableVec<usize, u32> =
            StorableVec::forced_import(Path::new("./vec"), Version::from(1), Compressed::YES)?;

        (0..21_u32).for_each(|v| {
            vec.push(v);
        });
        dbg!(vec.get(0)?);
        dbg!(vec.get(20)?);
        dbg!(vec.get(21)?);

        vec.flush()?;
    }

    {
        let mut vec: StorableVec<usize, u32> =
            StorableVec::forced_import(Path::new("./vec"), Version::from(1), Compressed::YES)?;

        dbg!(vec.get(0)?);
        dbg!(vec.get(0)?);
        dbg!(vec.get(1)?);
        dbg!(vec.get(2)?);
        dbg!(vec.get(20)?);
        dbg!(vec.get(20)?);
        dbg!(vec.get(0)?);

        vec.push(21);
        vec.push(22);
        dbg!(vec.get(20)?);
        dbg!(vec.get(21)?);
        dbg!(vec.get(22)?);
        dbg!(vec.get(23)?);

        vec.flush()?;
    }

    {
        let mut vec: StorableVec<usize, u32> =
            StorableVec::forced_import(Path::new("./vec"), Version::from(1), Compressed::YES)?;

        vec.enable_large_cache();

        dbg!(vec.get(0)?);
        dbg!(vec.get(20)?);
        dbg!(vec.get(21)?);
        dbg!(vec.get(22)?);

        vec.truncate_if_needed(14)?;

        dbg!(vec.get(0)?);
        dbg!(vec.get(5)?);
        dbg!(vec.get(20)?);

        vec.iter(|(_, v)| {
            dbg!(v);
            Ok(())
        })?;

        vec.iter_from(5, |(_, v)| {
            dbg!(v);
            Ok(())
        })?;

        dbg!(vec.collect_range(Some(-5), None)?);
    }

    Ok(())
}