Crate vecdb 

vecdb

High-performance mutable persistent vectors built on rawdb.

It features:

• Vec based API: push, update, truncate, delete by index
• Multiple variants: raw, compressed, computed
• Rollback via stamped change deltas
• Sparse deletions with holes
• Thread-safe with concurrent reads
• Blazing fast (benchmark)
• Persistence only on flush

It is not:

• A key-value store (consider fjall or redb)
• Suited for variable-sized types (String, Vec<T>, etc.)

Install

cargo add vecdb

Usage

use vecdb::{AnyStoredVec, Database, GenericStoredVec, RawVec, Result, Version};

fn main() -> Result<()> {
    // create
    let temp_dir = tempfile::TempDir::new()?;
    let db = Database::open(temp_dir.path())?;
    let mut vec: RawVec<usize, u64> = RawVec::import(&db, "vec", Version::ONE)?;

    // push
    for i in 0..1_000_000 {
        vec.push(i);
    }

    // flush
    vec.flush()?;
    db.flush()?;

    // read (sequential)
    let mut sum = 0u64;
    for value in vec.iter()? {
        sum = sum.wrapping_add(value);
    }

    // read (random)
    let indices: Vec<usize> = vec![500, 1000, 10];
    let reader = vec.create_reader();
    for i in indices {
        if let Ok(value) = vec.read_at(i, &reader) {
            sum = sum.wrapping_add(value);
        }
    }

    Ok(())
}

Constraints

Data must be fixed-size types: numbers, fixed arrays, structs with #[repr(C)].

Compression via Pcodec works for numeric types only.

When to use it

• Need to store Vecs on disk
• Append-only or append-mostly workloads
• Need very high read speeds
• Space-efficient storage for numeric data
• Sparse deletions without reindexing
• Rollback without full snapshots

Examples

See examples/ for usage.

Structs

CompressedVec

Compressed storage vector using Pcodec for lossless numerical compression.

Database

Memory-mapped database with dynamic space allocation and hole punching.

EagerVec

Stored vector with eager computation methods for deriving values from other vectors.

Exit

Graceful shutdown coordinator for ensuring data consistency during program exit.

ImportOptions

Options for importing or creating stored vectors.

LazyVecFrom1

Lazily computed vector deriving values from one source vector.

LazyVecFrom2

Lazily computed vector deriving values from two source vectors.

LazyVecFrom3

Lazily computed vector deriving values from three source vectors.

RawVec

Raw storage vector that stores values as-is without compression.

Reader

Zero-copy reader for accessing region data from memory-mapped storage.

Stamp

Marker for tracking when data was last modified.

Version

Version tracking for data schema and computed values.

Enums

Computation

Computation strategy for derived vectors.

ComputedVec

Enum wrapper for computed vectors, supporting both eager and lazy computation strategies.

Error

Error types for vecdb operations.

Format

Storage format selection for stored vectors.

RawDBError

Error types for rawdb operations.

StoredVec

Enum wrapper for stored vectors, supporting both raw and compressed formats.

Constants

PAGE_SIZE

SEPARATOR

Traits

AnyCloneableIterableVec

Trait for iterable vectors that can be cloned as trait objects.

AnyCollectableVec

Type-erased trait for collectable vectors.

AnyIterableVec

Trait for vectors that can be iterated.

AnyStoredIterableVec

Trait combining stored and iterable vector capabilities.

AnyStoredVec

Trait for stored vectors that persist data to disk (as opposed to lazy computed vectors).

AnyVec

Common trait for all vectors providing metadata and utility methods.

AsInnerSlice

CheckedSub

CollectableVec

Trait for vectors that can be collected into standard Rust collections with range support.

FromCoarserIndex

FromInnerSlice

GenericStoredVec

PrintableIndex

StoredCompressed

StoredIndex

StoredRaw

TransparentStoredCompressed

VecIterator

Base trait for vector iterators with positioning capabilities.

VecIteratorExtended

Extended vector iterator with type-safe index operations.

Functions

i64_to_usize

Converts an i64 index to usize, supporting negative indexing. Negative indices count from the end.

Type Aliases

AnyBoxedIterableVec

Type alias for boxed cloneable iterable vectors.

BoxedVecIterator

Type alias for boxed vector iterators.

ComputedVecFrom1

ComputedVecFrom2

ComputedVecFrom3

Result

Derive Macros

StoredCompressed