Crate dataspool_rs 

DataSpool: Efficient Data Bundling System

DataSpool eliminates filesystem overhead by concatenating multiple items (cards, images, etc.) into a single indexed file with SQLite-based metadata.

Features

• Spool Format: Single file with byte-offset index
• Vector Database: SQLite-based embeddings and metadata
• Random Access: Direct byte offset seeks, no scanning
• Multiple Variants: Cards, Images (future), Binary blobs

Format

.spool file:
  [Magic: SP01][Version: 1]
  [Card Count: u32][Index Offset: u64]
  [Card 0 data...][Card 1 data...][Card N data...]
  [Index: offset0, len0, offset1, len1, ...]

.db file (SQLite):
  - documents table (id, file_path, source, metadata, spool_offset, spool_length)
  - embeddings table (doc_id, vector BLOB)

Re-exports

pub use error::DataSpoolError;

pub use error::Result;

pub use spool::SpoolBuilder;

pub use spool::SpoolEntry;

pub use spool::SpoolReader;

pub use persistent_store::DocumentRef;

pub use persistent_store::FileSearchResult;

pub use persistent_store::PersistentVectorStore;

Modules

error

Error types for DataSpool

persistent_store

Persistent Vector Store with SQLite

spool

Data Spooling: Stitch BP Cards into Single File