lucistore 4.0.2

Shared persistence, sync and shard infrastructure for lucivy and friends
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

# lucistore

Shared persistence, sync and shard infrastructure for
[lucivy](https://github.com/L-Defraiteur/lucivy) and its friend crates
(`lucivy-core` full-text index, `sparse-vector` sparse index). Depends on
`serde` only.

## What it provides

- **`BlobStore`** — the storage contract: `load`, `save`, `delete`,
  `exists`, `list`, plus optional `blob_len` and `load_range` for backends
  that can answer cheaply (SQL `LENGTH`/`SUBSTRING`, S3 HEAD / ranged GET).
  `MemBlobStore` for tests, `impl BlobStore for Arc<T>` so `Arc<dyn BlobStore>`
  works everywhere.
- **`ShardStorage`** — where an index's shards live: `FsShardStorage`
  (directories) or `BlobShardStorage<S: BlobStore>` (blobs are the source of
  truth, a local cache holds the mmaps).
- **`ShardRouter`** — document-to-shard routing shared by every sharded
  index: round-robin (`balance_weight = 1.0`) or locality-aware (lower
  values co-locate documents sharing tokens), `node_id → shard` map,
  serializable, `resync` from the shards after a crash.
- **Snapshots and deltas** — `snapshot` (LUCE: every shard), `delta` (LUCID:
  one shard) and `delta_sharded` (LUCIDS: only the shards that changed,
  with per-shard versions), all as versioned binary buffers.
- **`sync_server`** — server-side version history per shard: computes the
  delta a client needs, or signals that it is too far behind and needs a
  full snapshot. Engine-agnostic.
- **`blob_cache`** — "the store holds, the mmap serves": blobs materialised
  into a local cache directory, write-through, cleaned up on drop.
- **`version`, `binary`, `fs_utils`** — deterministic version hashes of a
  manifest, length-prefixed binary helpers, filesystem utilities.

## Using it

An index that wants to be persisted anywhere implements its files through a
`BlobStore` and its layout through a `ShardStorage`; the rest (routing,
snapshots, deltas) comes for free and is identical across index types.

```rust
use std::sync::Arc;
use lucistore::blob_store::{BlobStore, MemBlobStore};
use lucistore::shard_router::ShardRouter;

let store: Arc<dyn BlobStore> = Arc::new(MemBlobStore::new());
store.save("my_index", "hello.bin", b"payload")?;
assert_eq!(store.load("my_index", "hello.bin")?, b"payload");

let mut router = ShardRouter::new(4);
let shard = router.route(&[ShardRouter::hash_token("rust"), ShardRouter::hash_token("search")]);
assert!(shard < 4);
# Ok::<(), std::io::Error>(())
```

## License

MIT.