Struct MmapStorage 

pub struct MmapStorage { /* private fields */ }

Memory-mapped file storage for vectors.

Uses a combination of an index file (ID -> offset) and a data file (raw vectors). Also implements a simple WAL for durability.

Implementations

impl MmapStorage

pub fn new<P: AsRef<Path>>(path: P, dimension: usize) -> Result<Self>

Creates a new MmapStorage or opens an existing one.

Arguments

• path - Directory to store data
• dimension - Vector dimension

Errors

Returns an error if file operations fail.

pub fn reserve_capacity(&mut self, vector_count: usize) -> Result<()>

Pre-allocates storage capacity for a known number of vectors.

Call this before bulk imports to avoid blocking resize operations during insertion. This is especially useful when the final dataset size is known in advance.

P2 Optimization

This allows users to pre-allocate once and avoid all resize locks during bulk import operations.

Arguments

• vector_count - Expected number of vectors to store

Example

// Pre-allocate for 1 million vectors before bulk import
storage.reserve_capacity(1_000_000)?;

Errors

Returns an error if file operations fail.

pub fn metrics(&self) -> &StorageMetrics

Returns a reference to the storage metrics.

P0 Audit: Latency Monitoring

Use this to monitor ensure_capacity latency, especially P99. High P99 latency indicates “stop-the-world” pauses during mmap resizes.

Example

let storage = MmapStorage::new(path, 768)?;
// ... perform operations ...
let stats = storage.metrics().ensure_capacity_latency_stats();
if stats.p99_exceeds(Duration::from_millis(100)) {
    warn!("High P99 latency detected: {:?}", stats.p99());
}

pub fn compact(&mut self) -> Result<usize>

Compacts the storage by rewriting only active vectors.

This reclaims disk space from deleted vectors by:

1. Writing all active vectors to a new temporary file
2. Atomically replacing the old file with the new one

TS-CORE-004: Storage Compaction

This operation is quasi-atomic via rename() for crash safety. Reads remain available during compaction (copy-on-write pattern).

Returns

The number of bytes reclaimed.

Errors

Returns an error if file operations fail.

pub fn fragmentation_ratio(&self) -> f64

Returns the fragmentation ratio (0.0 = no fragmentation, 1.0 = 100% fragmented).

Use this to decide when to trigger compaction. A ratio > 0.3 (30% fragmentation) is a good threshold.

pub fn retrieve_ref(&self, id: u64) -> Result<Option<VectorSliceGuard<'_>>>

Retrieves a vector by ID without copying (zero-copy).

Returns a guard that provides direct access to the mmap’d data. This is significantly faster than retrieve() for read-heavy workloads as it eliminates heap allocation and memory copy.

Arguments

• id - The vector ID to retrieve

Returns

• Ok(Some(guard)) - Guard providing zero-copy access to the vector
• Ok(None) - Vector with this ID doesn’t exist
• Err(...) - I/O error (e.g., corrupted data)

Example

let guard = storage.retrieve_ref(id)?.unwrap();
let slice: &[f32] = guard.as_ref();
// Use slice directly - no allocation occurred

Performance

Compared to retrieve():

• No heap allocation - data accessed directly from mmap
• No memcpy - pointer arithmetic only
• Lock held - guard must be dropped to release read lock

Errors

Returns an error if the stored offset is out of bounds (corrupted index).

Trait Implementations

impl VectorStorage for MmapStorage

fn store(&mut self, id: u64, vector: &[f32]) -> Result<()>

Stores a vector with the given ID.

fn store_batch(&mut self, vectors: &[(u64, &[f32])]) -> Result<usize>

Stores multiple vectors in a single batch operation.

fn retrieve(&self, id: u64) -> Result<Option<Vec<f32>>>

Retrieves a vector by ID.

fn delete(&mut self, id: u64) -> Result<()>

Deletes a vector by ID.

fn flush(&mut self) -> Result<()>

Flushes pending writes to disk.

fn len(&self) -> usize

Returns the number of vectors stored.

fn ids(&self) -> Vec<u64>

Returns all stored IDs.

fn is_empty(&self) -> bool

Returns true if the storage is empty.