Expand description
§vecdb
A K.I.S.S. index-value storage engine that provides persistent, type-safe vector storage with compression and computation capabilities.
§What is vecdb?
vecdb is a high-level vector storage engine built on seqdb that provides persistent vector-like data structures. It supports multiple storage formats and computation strategies for different performance and space requirements.
§Key Features
- Multiple storage variants: Raw, compressed, lazy, eager, and computed vectors
- Advanced compression: Uses Pcodec for numerical data compression
- Type safety: Generic storage with zero-copy access
- Versioning system: Change tracking and rollback support
- Hole management: Efficient sparse data handling
- Thread-safe: Concurrent read operations
§Storage Variants
§RawVec - Uncompressed Storage
Fast, direct storage without compression.
use vecdb::{Database, RawVec, Version};
use std::path::Path;
let db = Database::open(Path::new("data"))?;
let mut vec: RawVec<usize, u32> = RawVec::forced_import(&db, "numbers", Version::TWO)?;
// Basic operations
vec.push(42);
vec.push(84);
// Reading with zero-copy when possible
let reader = vec.create_static_reader();
let value = vec.get_or_read(0, &reader)?.unwrap();
assert_eq!(*value, 42);
drop(reader);
// Updates and deletions
vec.update(0, 100)?;
let removed = vec.take(1, &vec.create_static_reader())?; // Creates a hole
vec.flush()?;§CompressedVec - Space-Efficient Storage
Automatic compression using Pcodec for numerical data.
use vecdb::{Database, CompressedVec, Version};
let db = Database::open(Path::new("data"))?;
let mut vec: CompressedVec<usize, f64> =
CompressedVec::forced_import(&db, "measurements", Version::TWO)?;
// Same API as RawVec but with automatic compression
for i in 0..1000 {
vec.push(i as f64 * 3.14159);
}
vec.flush()?; // Data compressed on flush
// Reading transparently decompresses
let reader = vec.create_static_reader();
let value = vec.get_or_read(500, &reader)?;§ComputedVec - Derived Data
On-demand or pre-computed vectors derived from other vectors.
use vecdb::{ComputedVecFrom1, Computation, Format};
// Source data
let mut source: RawVec<usize, f64> = RawVec::forced_import(&db, "source", Version::TWO)?;
source.push(2.0);
source.push(3.0);
source.flush()?;
// Computed vector (squares the source values)
let computed = ComputedVecFrom1::forced_import_or_init_from_1(
&db,
"squares",
Version::TWO,
Computation::Eager,
Format::Compressed,
source.boxed_iter(),
|_index, iter| {
iter.get(_index).map(|(_, value)| value.as_ref() * value.as_ref())
},
)?;§Core Operations
§Basic Vector Operations
let mut vec: RawVec<usize, i32> = RawVec::forced_import(&db, "data", Version::TWO)?;
// Adding elements
vec.push(10);
vec.push(20);
vec.push(30);
// Reading elements
let reader = vec.create_static_reader();
let value = vec.get_or_read(1, &reader)?;
drop(reader);
// Updating elements
vec.update(0, 15)?;
// Removing elements (creates holes)
let removed = vec.take(1, &vec.create_static_reader())?;
// Fill holes when adding new data
let new_index = vec.fill_first_hole_or_push(25)?;
// Persistence
vec.flush()?;§Collection and Iteration
// Collect all values (skipping holes)
let values: Vec<i32> = vec.collect()?;
// Collect including holes as Option<T>
let with_holes: Vec<Option<i32>> = vec.collect_holed()?;
// Iterator support
for (index, value) in &vec {
println!("vec[{}] = {}", index, value.as_ref());
}
// Range iteration
let last_5: Vec<i32> = vec.collect_signed_range(Some(-5), None)?;§Version Control
use vecdb::Stamp;
// Save with version stamp
vec.stamped_flush(Stamp::new(42))?;
// Rollback to previous version
vec.rollback_stamp(Stamp::new(41))?;§Type Requirements
§Index Types
Must implement StoredIndex trait (built-in for usize, u32, u64, etc.).
§Value Types
For RawVec (StoredRaw):
FromBytes+IntoBytes(zerocopy traits)Clone+Copy+Debug+Send+Sync
For CompressedVec (StoredCompressed):
- All
StoredRawrequirements - Numerical types:
u8,u16,u32,u64,i8,i16,i32,i64,f32,f64 - Custom types via
#[derive(StoredCompressed)]with vecdb_derive
§Performance Characteristics
| Operation | RawVec | CompressedVec | ComputedVec |
|---|---|---|---|
| Random Read | O(1) | O(page_size) | O(computation) |
| Sequential Read | Fastest | Fast | Variable |
| Write | Fastest | Fast | N/A |
| Space Usage | 1.0x | 0.1x - 0.5x | Variable |
§Best Practices
- Choose the right variant: RawVec for speed, CompressedVec for space, ComputedVec for derived data
- Manage readers: Create readers for read operations, drop before mutations
- Batch operations: Flush once after multiple operations for better performance
- Handle concurrent access: Multiple readers are safe, coordinate writers externally
§Use Cases
- Time-series data: Compressed storage of sensor readings
- Analytics: Derived computations from base datasets
- Caching: Persistent memoization of expensive computations
- Scientific computing: Large numerical datasets with compression
§Error Handling
use vecdb::{Error, Result};
match vec.get_or_read(index, &reader) {
Ok(Some(value)) => println!("Found: {:?}", value),
Ok(None) => println!("Hole at index {}", index),
Err(Error::IndexTooHigh) => println!("Index out of bounds"),
Err(Error::DifferentVersion { .. }) => println!("Version mismatch"),
Err(e) => println!("Other error: {}", e),
}§Integration
Built on seqdb for:
- Memory-mapped file access
- Page-aligned storage
- Cross-platform file locking
- Dynamic space management
This README was generated by Claude Code
§Examples
§Raw
use std::{borrow::Cow, collections::BTreeSet, fs, path::Path};
use vecdb::{
AnyStoredVec, AnyVec, CollectableVec, Database, GenericStoredVec, RawVec, Stamp, VecIterator,
Version,
};
#[allow(clippy::upper_case_acronyms)]
type VEC = RawVec<usize, u32>;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let _ = fs::remove_dir_all("raw");
let version = Version::TWO;
let database = Database::open(Path::new("raw"))?;
let mut options = (&database, "vec", version).into();
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
(0..21_u32).for_each(|v| {
vec.push(v);
});
let mut iter = vec.into_iter();
assert!(iter.get(0) == Some(Cow::Borrowed(&0)));
assert!(iter.get(1) == Some(Cow::Borrowed(&1)));
assert!(iter.get(2) == Some(Cow::Borrowed(&2)));
assert!(iter.get(20) == Some(Cow::Borrowed(&20)));
assert!(iter.get(21).is_none());
drop(iter);
vec.flush()?;
assert!(vec.header().stamp() == Stamp::new(0));
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
vec.mut_header().update_stamp(Stamp::new(100));
assert!(vec.header().stamp() == Stamp::new(100));
let mut iter = vec.into_iter();
assert!(iter.get(0) == Some(Cow::Borrowed(&0)));
assert!(iter.get(1) == Some(Cow::Borrowed(&1)));
assert!(iter.get(2) == Some(Cow::Borrowed(&2)));
assert!(iter.get(3) == Some(Cow::Borrowed(&3)));
assert!(iter.get(4) == Some(Cow::Borrowed(&4)));
assert!(iter.get(5) == Some(Cow::Borrowed(&5)));
assert!(iter.get(20) == Some(Cow::Borrowed(&20)));
assert!(iter.get(20) == Some(Cow::Borrowed(&20)));
assert!(iter.get(0) == Some(Cow::Borrowed(&0)));
drop(iter);
vec.push(21);
vec.push(22);
assert!(vec.stored_len() == 21);
assert!(vec.pushed_len() == 2);
assert!(vec.len() == 23);
let mut iter = vec.into_iter();
assert!(iter.get(20) == Some(Cow::Borrowed(&20)));
assert!(iter.get(21) == Some(Cow::Borrowed(&21)));
assert!(iter.get(22) == Some(Cow::Borrowed(&22)));
assert!(iter.get(23).is_none());
drop(iter);
vec.flush()?;
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
assert!(vec.header().stamp() == Stamp::new(100));
assert!(vec.stored_len() == 23);
assert!(vec.pushed_len() == 0);
assert!(vec.len() == 23);
let mut iter = vec.into_iter();
assert!(iter.get(0) == Some(Cow::Borrowed(&0)));
assert!(iter.get(20) == Some(Cow::Borrowed(&20)));
assert!(iter.get(21) == Some(Cow::Borrowed(&21)));
assert!(iter.get(22) == Some(Cow::Borrowed(&22)));
drop(iter);
vec.truncate_if_needed(14)?;
assert_eq!(vec.stored_len(), 14);
assert_eq!(vec.pushed_len(), 0);
assert_eq!(vec.len(), 14);
let mut iter = vec.into_iter();
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
assert_eq!(iter.get(5), Some(Cow::Borrowed(&5)));
assert_eq!(iter.get(20), None);
drop(iter);
assert_eq!(
vec.collect_signed_range(Some(-5), None)?,
vec![9, 10, 11, 12, 13]
);
vec.push(vec.len() as u32);
assert_eq!(
VecIterator::last(vec.into_iter()),
Some((14, Cow::Borrowed(&14)))
);
assert_eq!(
vec.into_iter()
.map(|(_, v)| v.into_owned())
.collect::<Vec<_>>(),
vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
);
vec.flush()?;
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
assert_eq!(
VecIterator::last(vec.into_iter()),
Some((14, Cow::Borrowed(&14)))
);
assert_eq!(
vec.into_iter()
.map(|(_, v)| v.into_owned())
.collect::<Vec<_>>(),
vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
);
vec.reset()?;
assert_eq!(vec.pushed_len(), 0);
assert_eq!(vec.stored_len(), 0);
assert_eq!(vec.len(), 0);
(0..21_u32).for_each(|v| {
vec.push(v);
});
assert_eq!(vec.pushed_len(), 21);
assert_eq!(vec.stored_len(), 0);
assert_eq!(vec.len(), 21);
let mut iter = vec.into_iter();
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
assert_eq!(iter.get(20), Some(Cow::Borrowed(&20)));
assert!(iter.get(21).is_none());
drop(iter);
let reader = vec.create_static_reader();
assert_eq!(vec.take(10, &reader)?, Some(10));
assert_eq!(vec.holes(), &BTreeSet::from([10]));
assert!(vec.get_or_read(10, &reader)?.is_none());
drop(reader);
vec.flush()?;
assert!(vec.holes() == &BTreeSet::from([10]));
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
assert!(vec.holes() == &BTreeSet::from([10]));
let reader = vec.create_static_reader();
assert!(vec.get_or_read(10, &reader)?.is_none());
drop(reader);
vec.update(10, 10)?;
vec.update(0, 10)?;
let reader = vec.create_static_reader();
assert_eq!(vec.holes(), &BTreeSet::new());
assert_eq!(vec.get_or_read(0, &reader)?, Some(Cow::Borrowed(&10)));
assert_eq!(vec.get_or_read(10, &reader)?, Some(Cow::Borrowed(&10)));
drop(reader);
vec.flush()?;
}
options = options.with_saved_stamped_changes(10);
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
vec.truncate_if_needed(10)?;
let reader = vec.create_static_reader();
vec.take(5, &reader)?;
vec.update(3, 5)?;
vec.push(21);
drop(reader);
assert_eq!(
vec.collect_holed()?,
vec![
Some(10),
Some(1),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21)
]
);
vec.stamped_flush_with_changes(Stamp::new(1))?;
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
assert_eq!(vec.collect()?, vec![10, 1, 2, 5, 4]);
let reader = vec.create_static_reader();
vec.take(0, &reader)?;
vec.update(1, 5)?;
vec.push(5);
vec.push(6);
vec.push(7);
drop(reader);
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
vec.stamped_flush_with_changes(Stamp::new(2))?;
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
vec.rollback()?;
assert_eq!(vec.stamp(), Stamp::new(1));
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(
vec.collect_holed()?,
vec![
Some(10),
Some(1),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21)
]
);
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
vec.rollback()?;
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
vec.stamped_flush(Stamp::new(0))?;
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
vec.truncate_if_needed(10)?;
let reader = vec.create_static_reader();
vec.take(5, &reader)?;
vec.update(3, 5)?;
vec.push(21);
drop(reader);
assert_eq!(
vec.collect_holed()?,
vec![
Some(10),
Some(1),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21)
]
);
vec.stamped_flush_with_changes(Stamp::new(1))?;
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
assert_eq!(vec.collect()?, vec![10, 1, 2, 5, 4]);
let reader = vec.create_static_reader();
vec.take(0, &reader)?;
vec.update(1, 5)?;
vec.push(5);
vec.push(6);
vec.push(7);
drop(reader);
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
vec.stamped_flush_with_changes(Stamp::new(2))?;
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
vec.rollback_before(Stamp::new(1))?;
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
vec.stamped_flush(Stamp::new(0))?;
vec.truncate_if_needed(10)?;
let reader = vec.create_static_reader();
vec.take(5, &reader)?;
vec.update(3, 5)?;
vec.push(21);
drop(reader);
let reader = vec.create_static_reader();
vec.take(0, &reader)?;
vec.update(1, 5)?;
vec.push(5);
vec.push(6);
vec.push(7);
drop(reader);
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
dbg!(("0", vec.prev_holes(), vec.updated()));
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
vec.truncate_if_needed(10)?;
let reader = vec.create_static_reader();
vec.take(5, &reader)?;
vec.update(3, 5)?;
vec.push(21);
drop(reader);
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
vec.stamped_flush_with_changes(Stamp::new(1))?;
assert_eq!(vec.stamp(), Stamp::new(1));
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
let reader = vec.create_static_reader();
vec.take(0, &reader)?;
vec.update(1, 5)?;
vec.push(5);
vec.push(6);
vec.push(7);
drop(reader);
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
vec.stamped_flush_with_changes(Stamp::new(2))?;
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
vec.rollback_before(Stamp::new(1))?;
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
assert_eq!(vec.stamp(), Stamp::new(0));
vec.truncate_if_needed(10)?;
let reader = vec.create_static_reader();
vec.take(5, &reader)?;
vec.update(3, 5)?;
vec.push(21);
drop(reader);
let reader = vec.create_static_reader();
vec.take(0, &reader)?;
vec.update(1, 5)?;
vec.push(5);
vec.push(6);
vec.push(7);
drop(reader);
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
assert_eq!(vec.stamp(), Stamp::new(0));
vec.stamped_flush_with_changes(Stamp::new(2))?;
assert_eq!(vec.stamp(), Stamp::new(2));
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
dbg!("-----------------------------------------------------------------------");
vec.rollback_before(Stamp::new(1))?;
dbg!((
vec.prev_stored_len(),
vec.stored_len(),
vec.real_stored_len(),
vec.prev_pushed(),
vec.pushed(),
vec.prev_updated(),
vec.updated(),
vec.prev_holes(),
vec.holes(),
));
assert_eq!(vec.stamp(), Stamp::new(0));
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
vec.stamped_flush_with_changes(Stamp::new(0))?;
let vec: VEC = RawVec::forced_import_with(options)?;
dbg!(("0", vec.prev_holes(), vec.updated()));
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
}
{
let mut vec: VEC = RawVec::forced_import_with(options)?;
dbg!(("0", vec.stamp(), vec.stored_len()));
vec.truncate_if_needed(10)?;
let reader = vec.create_static_reader();
vec.take(5, &reader)?;
vec.update(3, 5)?;
vec.push(21);
drop(reader);
vec.stamped_flush_with_changes(Stamp::new(1))?;
assert_eq!(vec.stamp(), Stamp::new(1));
dbg!(("1", vec.stamp(), vec.stored_len()));
let reader = vec.create_static_reader();
vec.take(0, &reader)?;
vec.update(1, 5)?;
vec.push(5);
vec.push(6);
vec.push(7);
drop(reader);
vec.stamped_flush_with_changes(Stamp::new(2))?;
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
vec.rollback_before(Stamp::new(1))?;
dbg!(("roll", vec.stamp(), vec.stored_len()));
assert_eq!(vec.stamp(), Stamp::new(0));
vec.truncate_if_needed(10)?;
let reader = vec.create_static_reader();
vec.take(5, &reader)?;
vec.update(3, 5)?;
vec.push(21);
drop(reader);
let reader = vec.create_static_reader();
vec.take(0, &reader)?;
vec.update(1, 5)?;
vec.push(5);
vec.push(6);
vec.push(7);
drop(reader);
assert_eq!(
vec.collect_holed()?,
vec![
None,
Some(5),
Some(2),
Some(5),
Some(4),
None,
Some(6),
Some(7),
Some(8),
Some(9),
Some(21),
Some(5),
Some(6),
Some(7)
]
);
dbg!(1);
vec.rollback_before(Stamp::new(1))?;
dbg!(1);
assert_eq!(vec.stamp(), Stamp::new(0));
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
vec.stamped_flush_with_changes(Stamp::new(0))?;
let vec: VEC = RawVec::forced_import_with(options)?;
dbg!(("0", vec.prev_holes(), vec.updated()));
assert_eq!(
vec.collect()?,
vec![
10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
}
Ok(())
}§Compressed
use std::{borrow::Cow, collections::BTreeSet, fs, path::Path};
use vecdb::{
AnyStoredVec, AnyVec, CollectableVec, CompressedVec, Database, GenericStoredVec, Stamp,
VecIterator, Version,
};
#[allow(clippy::upper_case_acronyms)]
type VEC = CompressedVec<usize, u32>;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let _ = fs::remove_dir_all("compressed");
let version = Version::TWO;
let database = Database::open(Path::new("compressed"))?;
let options = (&database, "vec", version).into();
{
let mut vec: VEC = CompressedVec::forced_import_with(options)?;
(0..21_u32).for_each(|v| {
vec.push(v);
});
let mut iter = vec.into_iter();
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
assert_eq!(iter.get(1), Some(Cow::Borrowed(&1)));
assert_eq!(iter.get(2), Some(Cow::Borrowed(&2)));
assert_eq!(iter.get(20), Some(Cow::Borrowed(&20)));
assert_eq!(iter.get(21), None);
drop(iter);
vec.flush()?;
assert_eq!(vec.header().stamp(), Stamp::new(0));
}
{
let mut vec: VEC = CompressedVec::forced_import_with(options)?;
vec.mut_header().update_stamp(Stamp::new(100));
assert!(vec.header().stamp() == Stamp::new(100));
let mut iter = vec.into_iter();
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
assert_eq!(iter.get(1), Some(Cow::Borrowed(&1)));
assert_eq!(iter.get(2), Some(Cow::Borrowed(&2)));
assert_eq!(iter.get(3), Some(Cow::Borrowed(&3)));
assert_eq!(iter.get(4), Some(Cow::Borrowed(&4)));
assert_eq!(iter.get(5), Some(Cow::Borrowed(&5)));
assert_eq!(iter.get(20), Some(Cow::Borrowed(&20)));
assert_eq!(iter.get(20), Some(Cow::Borrowed(&20)));
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
drop(iter);
vec.push(21);
vec.push(22);
assert_eq!(vec.stored_len(), 21);
assert_eq!(vec.pushed_len(), 2);
assert_eq!(vec.len(), 23);
let mut iter = vec.into_iter();
assert_eq!(iter.get(20), Some(Cow::Borrowed(&20)));
assert_eq!(iter.get(21), Some(Cow::Borrowed(&21)));
assert_eq!(iter.get(22), Some(Cow::Borrowed(&22)));
assert_eq!(iter.get(23), None);
drop(iter);
vec.flush()?;
}
{
let mut vec: VEC = CompressedVec::forced_import_with(options)?;
assert_eq!(vec.header().stamp(), Stamp::new(100));
assert_eq!(vec.stored_len(), 23);
assert_eq!(vec.pushed_len(), 0);
assert_eq!(vec.len(), 23);
let mut iter = vec.into_iter();
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
assert_eq!(iter.get(20), Some(Cow::Borrowed(&20)));
assert_eq!(iter.get(21), Some(Cow::Borrowed(&21)));
assert_eq!(iter.get(22), Some(Cow::Borrowed(&22)));
drop(iter);
vec.truncate_if_needed(14)?;
assert_eq!(vec.stored_len(), 14);
assert_eq!(vec.pushed_len(), 0);
assert_eq!(vec.len(), 14);
let mut iter = vec.into_iter();
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
assert_eq!(iter.get(5), Some(Cow::Borrowed(&5)));
assert_eq!(iter.get(20), None);
drop(iter);
assert_eq!(
vec.collect_signed_range(Some(-5), None)?,
vec![9, 10, 11, 12, 13]
);
vec.push(vec.len() as u32);
assert_eq!(
VecIterator::last(vec.into_iter()),
Some((14, Cow::Borrowed(&14)))
);
vec.flush()?;
assert_eq!(
vec.into_iter()
.map(|(_, v)| v.into_owned())
.collect::<Vec<_>>(),
vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
);
}
{
let mut vec: VEC = CompressedVec::forced_import_with(options)?;
assert_eq!(
vec.into_iter()
.map(|(_, v)| v.into_owned())
.collect::<Vec<_>>(),
vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
);
let mut iter = vec.into_iter();
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
assert_eq!(iter.get(5), Some(Cow::Borrowed(&5)));
assert_eq!(iter.get(20), None);
drop(iter);
assert_eq!(
vec.collect_signed_range(Some(-5), None)?,
vec![10, 11, 12, 13, 14]
);
vec.reset()?;
assert_eq!(vec.pushed_len(), 0);
assert_eq!(vec.stored_len(), 0);
assert_eq!(vec.len(), 0);
(0..21_u32).for_each(|v| {
vec.push(v);
});
assert_eq!(vec.pushed_len(), 21);
assert_eq!(vec.stored_len(), 0);
assert_eq!(vec.len(), 21);
let mut iter = vec.into_iter();
assert_eq!(iter.get(0), Some(Cow::Borrowed(&0)));
assert_eq!(iter.get(20), Some(Cow::Borrowed(&20)));
assert_eq!(iter.get(21), None);
drop(iter);
vec.flush()?;
}
{
let mut vec: VEC = CompressedVec::forced_import_with(options)?;
assert_eq!(vec.pushed_len(), 0);
assert_eq!(vec.stored_len(), 21);
assert_eq!(vec.len(), 21);
let reader = vec.create_static_reader();
assert_eq!(vec.holes(), &BTreeSet::new());
assert_eq!(vec.get_or_read(0, &reader)?, Some(Cow::Borrowed(&0)));
assert_eq!(vec.get_or_read(10, &reader)?, Some(Cow::Borrowed(&10)));
drop(reader);
vec.flush()?;
}
{
let vec: VEC = CompressedVec::forced_import_with(options)?;
assert!(
vec.collect()?
== vec![
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
]
);
}
Ok(())
}Structs§
- Compressed
Vec - Database
- Eager
Vec - Exit
- Import
Options - Lazy
VecFrom1 - Lazy
VecFrom2 - Lazy
VecFrom3 - RawVec
- Reader
- Stamp
- Version
Enums§
Constants§
Traits§
- AnyCloneable
Iterable Vec - AnyCollectable
Vec - AnyIterable
Vec - AnyStored
Iterable Vec - AnyStored
Vec - AnyVec
- AsInner
Slice - Base
VecIterator - Checked
Sub - Collectable
Vec - From
Coarser Index - From
Inner Slice - Generic
Stored Vec - Printable
- Stored
Compressed - Stored
Index - Stored
Raw - Transparent
Stored Compressed - VecIterator