vecdb
A K.I.S.S. index-value storage engine that provides persistent, type-safe vector storage with compression and computation capabilities.
What is vecdb?
vecdb is a high-level vector storage engine built on seqdb that provides persistent vector-like data structures. It supports multiple storage formats and computation strategies for different performance and space requirements.
Key Features
- Multiple storage variants: Raw, compressed, lazy, eager, and computed vectors
- Advanced compression: Uses Pcodec for numerical data compression
- Type safety: Generic storage with zero-copy access
- Versioning system: Change tracking and rollback support
- Hole management: Efficient sparse data handling
- Thread-safe: Concurrent read operations
Storage Variants
RawVec - Uncompressed Storage
Fast, direct storage without compression.
use ;
use Path;
let db = open?;
let mut vec: = forced_import?;
// Basic operations
vec.push;
vec.push;
// Reading with zero-copy when possible
let reader = vec.create_static_reader;
let value = vec.get_or_read?.unwrap;
assert_eq!;
drop;
// Updates and deletions
vec.update?;
let removed = vec.take?; // Creates a hole
vec.flush?;
CompressedVec - Space-Efficient Storage
Automatic compression using Pcodec for numerical data.
use ;
let db = open?;
let mut vec: =
forced_import?;
// Same API as RawVec but with automatic compression
for i in 0..1000
vec.flush?; // Data compressed on flush
// Reading transparently decompresses
let reader = vec.create_static_reader;
let value = vec.get_or_read?;
ComputedVec - Derived Data
On-demand or pre-computed vectors derived from other vectors.
use ;
// Source data
let mut source: = forced_import?;
source.push;
source.push;
source.flush?;
// Computed vector (squares the source values)
let computed = forced_import_or_init_from_1?;
Core Operations
Basic Vector Operations
let mut vec: = forced_import?;
// Adding elements
vec.push;
vec.push;
vec.push;
// Reading elements
let reader = vec.create_static_reader;
let value = vec.get_or_read?;
drop;
// Updating elements
vec.update?;
// Removing elements (creates holes)
let removed = vec.take?;
// Fill holes when adding new data
let new_index = vec.fill_first_hole_or_push?;
// Persistence
vec.flush?;
Collection and Iteration
// Collect all values (skipping holes)
let values: = vec.collect?;
// Collect including holes as Option<T>
let with_holes: = vec.collect_holed?;
// Iterator support
for in &vec
// Range iteration
let last_5: = vec.collect_signed_range?;
Version Control
use Stamp;
// Save with version stamp
vec.stamped_flush?;
// Rollback to previous version
vec.rollback_stamp?;
Type Requirements
Index Types
Must implement StoredIndex
trait (built-in for usize
, u32
, u64
, etc.).
Value Types
For RawVec (StoredRaw
):
FromBytes
+IntoBytes
(zerocopy traits)Clone
+Copy
+Debug
+Send
+Sync
For CompressedVec (StoredCompressed
):
- All
StoredRaw
requirements - Numerical types:
u8
,u16
,u32
,u64
,i8
,i16
,i32
,i64
,f32
,f64
- Custom types via
#[derive(StoredCompressed)]
with vecdb_derive
Performance Characteristics
Operation | RawVec | CompressedVec | ComputedVec |
---|---|---|---|
Random Read | O(1) | O(page_size) | O(computation) |
Sequential Read | Fastest | Fast | Variable |
Write | Fastest | Fast | N/A |
Space Usage | 1.0x | 0.1x - 0.5x | Variable |
Best Practices
- Choose the right variant: RawVec for speed, CompressedVec for space, ComputedVec for derived data
- Manage readers: Create readers for read operations, drop before mutations
- Batch operations: Flush once after multiple operations for better performance
- Handle concurrent access: Multiple readers are safe, coordinate writers externally
Use Cases
- Time-series data: Compressed storage of sensor readings
- Analytics: Derived computations from base datasets
- Caching: Persistent memoization of expensive computations
- Scientific computing: Large numerical datasets with compression
Error Handling
use ;
match vec.get_or_read
Integration
Built on seqdb for:
- Memory-mapped file access
- Page-aligned storage
- Cross-platform file locking
- Dynamic space management
This README was generated by Claude Code