lucistore

Shared persistence, sync and shard infrastructure for lucivy and its friend crates (lucivy-core full-text index, sparse-vector sparse index). Depends on serde only.

What it provides

• BlobStore — the storage contract: load, save, delete, exists, list, plus optional blob_len and load_range for backends that can answer cheaply (SQL LENGTH/SUBSTRING, S3 HEAD / ranged GET). MemBlobStore for tests, impl BlobStore for Arc<T> so Arc<dyn BlobStore> works everywhere.
• ShardStorage — where an index's shards live: FsShardStorage (directories) or BlobShardStorage<S: BlobStore> (blobs are the source of truth, a local cache holds the mmaps).
• ShardRouter — document-to-shard routing shared by every sharded index: round-robin (balance_weight = 1.0) or locality-aware (lower values co-locate documents sharing tokens), node_id → shard map, serializable, resync from the shards after a crash.
• Snapshots and deltas — snapshot (LUCE: every shard), delta (LUCID: one shard) and delta_sharded (LUCIDS: only the shards that changed, with per-shard versions), all as versioned binary buffers.
• sync_server — server-side version history per shard: computes the delta a client needs, or signals that it is too far behind and needs a full snapshot. Engine-agnostic.
• blob_cache — "the store holds, the mmap serves": blobs materialised into a local cache directory, write-through, cleaned up on drop.
• version, binary, fs_utils — deterministic version hashes of a manifest, length-prefixed binary helpers, filesystem utilities.

Using it

An index that wants to be persisted anywhere implements its files through a BlobStore and its layout through a ShardStorage; the rest (routing, snapshots, deltas) comes for free and is identical across index types.

use std::sync::Arc;
use lucistore::blob_store::{BlobStore, MemBlobStore};
use lucistore::shard_router::ShardRouter;

let store: Arc<dyn BlobStore> = Arc::new(MemBlobStore::new());
store.save("my_index", "hello.bin", b"payload")?;
assert_eq!(store.load("my_index", "hello.bin")?, b"payload");

let mut router = ShardRouter::new(4);
let shard = router.route(&[ShardRouter::hash_token("rust"), ShardRouter::hash_token("search")]);
assert!(shard < 4);
# Ok::<(), std::io::Error>(())

License

MIT.