dataspool-rs 0.3.0

Efficient data bundling system with indexed .spool files and SQLite vector database
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

//! # DataSpool: Efficient Data Bundling System
//!
//! DataSpool eliminates filesystem overhead by concatenating multiple items
//! (cards, images, etc.) into a single indexed file with SQLite-based metadata.
//!
//! ## Features
//!
//! - **Spool Format**: Single file with byte-offset index
//! - **Vector Database**: SQLite-based embeddings and metadata
//! - **Random Access**: Direct byte offset seeks, no scanning
//! - **Multiple Variants**: Cards, Images (future), Binary blobs
//!
//! ## Format
//!
//! ```text
//! .spool file:
//!   [Magic: SP01][Version: 1]
//!   [Card Count: u32][Index Offset: u64]
//!   [Card 0 data...][Card 1 data...][Card N data...]
//!   [Index: offset0, len0, offset1, len1, ...]
//!
//! .db file (SQLite):
//!   - documents table (id, file_path, source, metadata, spool_offset, spool_length)
//!   - embeddings table (doc_id, vector BLOB)
//! ```

pub mod error;
pub mod spool;
pub mod persistent_store;

pub use error::{DataSpoolError, Result};
pub use spool::{SpoolBuilder, SpoolEntry, SpoolReader};
pub use persistent_store::{DocumentRef, FileSearchResult, PersistentVectorStore};