Skip to main content

Module sharded

Module sharded 

Source
Expand description

ShardedSparseHandle: N SparseHandles behind a ShardRouter and a pool of luciole actors — the sparse counterpart of lucivy_core’s ShardedHandle.

A dot product is local to a shard: no global statistics are needed to merge results, so a search is a scatter of the same (query, limit, filter) to every shard and a k-way merge of (id, score) by score.

Storage is abstracted by SparseShardStorage: a filesystem layout ({base}/shard_{i} plus root files) or a blob store (namespaces Sparse_{name}/shard_{i}, root files under Sparse_{name}), both built on lucistore’s shard storages.

Lifecycle contract, identical to the FTS handle: commit() persists every shard and the router; close() commits, stops the actors and makes the handle inert (every entry point answers "handle is closed"); drop_index() closes and destroys the storage.

Structs§

BlobSparseStorage
Blob storage: shard i is the SparseHandle namespace Sparse_{name}/shard_{i}, root files live under Sparse_{name}. The local cache under cache_base is disposable; the store is the truth.
FsSparseStorage
Filesystem storage: {base}/shard_{i}/ per shard, root files in {base}.
ShardedSparseConfig
Persisted as _sparse_config.json at the root of the storage.
ShardedSparseHandle

Traits§

SparseShardStorage
Where the shards and the root files of a sharded sparse index live.