Struct PayloadStore 

pub struct PayloadStore { /* private fields */ }

Payload storage engine

Architecture:

• Append-only payload file (payload.data)
• In-memory index: edge_id -> (offset, length, compression)
• Memory-mapped reads for zero-copy access
• Write-ahead log for crash recovery

Thread Safety:

• Reads: Lock-free via memory map (multiple concurrent readers)
• Writes: Serialized via RwLock (single writer)
• Index: Protected by RwLock

SCALABILITY:

• HashMap backend (default): Fast but uses ~50MB RAM per 1M payloads (32 bytes per entry + overhead). Suitable for < 10M traces.
• DiskHash backend: Disk-backed hash table scales to billions with minimal RAM. Recommended for 10M+ traces.

Usage:

use sochdb_storage::payload::{PayloadStore, IndexBackend};

// Default: in-memory HashMap
let store = PayloadStore::open("./data").unwrap();

// For 10M+ scale: disk-backed hash table
let store = PayloadStore::open_with_backend("./data", IndexBackend::DiskHash).unwrap();

Implementations

impl PayloadStore

pub fn open<P: AsRef<Path>>(data_dir: P) -> Result<Self>

Open or create a payload store with default HashMap index

pub fn open_with_backend<P: AsRef<Path>>( data_dir: P, backend: IndexBackend, ) -> Result<Self>

Open or create a payload store with specified index backend

pub fn append( &self, edge_id: u128, data: &[u8], compression: Option<CompressionType>, ) -> Result<(u64, u32, u8)>

Append a payload and return (offset, length, compression_type)

Compression Strategy:

• Small payloads (<1KB): No compression (overhead > benefit)
• Medium payloads (1KB-100KB): LZ4 (fast compression/decompression)
• Large payloads (>100KB): ZSTD (higher compression ratio)

This can be overridden by specifying compression_type explicitly.

DESKTOP APP FIX: Monitors memory growth and logs warnings at thresholds to prevent silent OOM in long-running sessions.

pub fn get(&self, edge_id: u128) -> Result<Option<Vec<u8>>>

Get a payload by edge ID

pub fn get_at_offset( &self, offset: u64, length: u32, compression: CompressionType, ) -> Result<Vec<u8>>

Get a payload at a specific offset

This is used when reading edges from SSTables that have embedded (offset, length, compression) tuples.

pub fn has_payload(&self, edge_id: u128) -> bool

Check if a payload exists for an edge

pub fn len(&self) -> usize

Get number of stored payloads

pub fn is_empty(&self) -> bool

Check if empty

pub fn save_index(&self) -> Result<()>

Save index to disk for fast recovery (delegates to backend)

pub fn stats(&self) -> PayloadStats

Get statistics

Trait Implementations

impl Drop for PayloadStore

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.