genegraph-storage 0.51.0

vector database: base Lance storage
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146

# genegraph-storage

A storage layer for graph-based vector databases.

Implements the **Lance format with an in-house writer/reader** (`lancefmt`, pinned to the Lance v2.1 spec) — no dependency on the official `lance` crate. Parquet interop paths and other formats are available via the `StorageBackend` trait.

Provided functionalities:
* `save_metadata`, `load_metadata`: a simple wrapper for all the data in the directory
* `save_*` / `load_*` dense matrices, sparse matrices, vectors, lambdas, indices
* **named collections** (RFC #81): schema-driven vector spaces (`save_vectors` / `load_vectors`, any column layout with one non-null `FixedSizeList` vector column) and first-class graph storage (`save_graph` / `load_graph`, weighted or topology-only, `u32`/`u64` node ids)
* vector-space ↔ graph linkage (a vector space references a graph collection through its `graph` property; resolved in one call with `Catalog::describe_vector_space`)
* **transactional generations**: atomic metadata commits (tmp + fsync + rename), `scoped_generation(n)` handles, generation listing/deletion for sweeps, reader pins
* **catalog contract** (`src/catalog.rs`): `TableDescriptor` + `Catalog` trait mirroring the Lance Namespace / Polaris Generic Table API shape, with `LocalRegistry` over the JSON metadata registry
* parquet interop (`save_dense_to_file` / `load_dense_from_file`)

A storage layer for:
* [`javelin-tui`](https://github.com/tuned-org-uk/javelin-tui): a graph-based vector database Text-Interface and
* [`arrowspace`](https://github.com/Mec-iS/arrowspace-rs): the next iteration of vector search

## Usage

```bash
cargo add genegraph_storage
```

Simple example (kept in sync with the compile-checked doc-test on `LanceStorageGraph`):

```rust
use genegraph_storage::lance_storage_graph::LanceStorageGraph;
use genegraph_storage::metadata::GeneMetadata;
use genegraph_storage::traits::backend::StorageBackend;
use genegraph_storage::traits::metadata::Metadata;
use smartcore::linalg::basic::arrays::{Array, Array2};
use smartcore::linalg::basic::matrix::DenseMatrix;

let base = std::env::temp_dir().join(format!("genegraph_doc_{}", std::process::id()));
let storage = LanceStorageGraph::new(
    base.to_string_lossy().to_string(),
    "doc_example".to_string(),
);

// some 2D data
let dense: Vec<Vec<f64>> = vec![vec![0.1, 0.4], vec![0.5, 0.2], vec![0.03, 0.8]];
let (nitems, nfeatures) = (dense.len(), dense[0].len());
let data = DenseMatrix::<f64>::from_iterator(
    dense.iter().flatten().copied(), nitems, nfeatures, 0);

// seed metadata FIRST to initialize the storage directory
let md = GeneMetadata::seed_metadata("doc_example", nitems, nfeatures, &storage)
    .await
    .unwrap();
let md_path = storage.save_metadata(&md).await.unwrap();

// your data is saved in an efficient Lance format
storage
    .save_dense("my_dataset", &data, &md_path)
    .await
    .unwrap();

// Loading back
let loaded = storage.load_dense("my_dataset").await.unwrap();
assert_eq!(loaded.shape(), (nitems, nfeatures));

std::fs::remove_dir_all(&base).ok();
```

Graphs are stored as edge-list collections and convert to CSR at the API boundary:

```rust
use genegraph_storage::graph::{GraphEdge, GraphWriteOptions, NodeIdWidth};

let edges = vec![GraphEdge::weighted(0, 1, 0.5), GraphEdge::weighted(1, 2, -0.25)];
storage.save_graph("my_graph", &edges, &md_path).await.unwrap();

let graph = storage.load_graph("my_graph").await.unwrap();
let csr = graph.to_csr().unwrap(); // sprs CsMat<f64>
```

Append-style writers get immutable, atomically-committed generations:

```rust
let gen = storage.scoped_generation(1); // artifacts at {logical}__g1_{key}.lance
```

### Concurrent writers and metadata safety

Every `save_*` call runs its metadata registry update through a per-path
commit actor, so concurrent tasks inside one process never lose updates.
Downstream code that performs its **own** metadata read-modify-write cycles
must use the same serialization. Two levels are public:

- **In-process** — `commit::with_commit_actor(metadata_path, cycle)`
  serializes your cycle against the registry paths of the same metadata
  file.
- **Cross-process** — the commit actor is per-process only. Independent
  processes (e.g. separate CLI invocations) take an advisory lock file
  around the whole cycle, resolved through the blessed convention
  `commit::lock_file_for_metadata(metadata_path)` (`{metadata-stem}.lock`
  next to the metadata file). The composed recipe — file lock held across
  the awaited actor cycle — is
  `commit::with_metadata_file_lock(metadata_path, cycle)`:

```rust
use genegraph_storage::commit::with_metadata_file_lock;

let metadata_path = std::path::Path::new("base/ds__g1_metadata.json");
with_metadata_file_lock(metadata_path, || async {
    // load → mutate → publish here: serialized across processes *and*
    // against in-process save_* cycles
    Ok(())
})
.await
.unwrap();
```

For consumers whose whole cycle is synchronous,
`commit::with_file_lock` takes the same lock file around a sync closure —
it serializes across processes but does not compose with the in-process
actor. Every writer of a given metadata file must take the same lock file
before mutating it — arbitration is only as strong as the convention.

**Operational note.** The flock wait is unbounded and runs through
`spawn_blocking`: a parked waiter cannot be aborted, and many long-lived
waiters can exhaust the runtime's blocking-thread capacity. That is a
sound trade for metadata commits *if* cycles stay short, contention is
normally brief, nothing under the lock does lengthy compute/network I/O
or waits indefinitely, and you understand shutdown behavior with a stuck
holder (blocked `spawn_blocking` tasks are abandoned by
`Runtime::shutdown_timeout`, awaited by `shutdown_background`; process
exit always releases the flock). If waits could be prolonged or numerous,
prefer a dedicated lock-management thread, an explicit timeout or
cancellation strategy, or a storage system with transactional
coordination.

## Lance format

The default build runs the in-house Lance v2.1 implementation (`lancefmt`) for all `StorageBackend` I/O: manifest with inline Overwrite transactions, txn files, version hints, MiniBlock pages with Flat / InlineBitpacking / FixedSizeList value compression. Encodings outside the supported subset are rejected with `StorageError::UnsupportedFormat` (never guessed). Interop conformance is fixture-based: golden fixtures written by the official lance crate are read back by the in-house reader's suite.

## Extending and traits

Every custom definition of a Lance database (store or manifold or data-cube) should implement the `Metadata` trait (or reuse `GeneMetadata`) and the `StorageBackend` trait like `LanceStorageGraph` does with `GeneMetadata` and `LanceStorage`. Collections additionally surface through the `Catalog` trait (`LocalRegistry` over `GeneMetadata`).

Traits in `traits` module can also be reused to implement other formats. Other formats can use `StorageBackend` to implement similar child-traits alike to `LanceStorage`. Then if matched with a custom `Metadata` instance can make a database, so every database is simply a `StorageBackend + Metadata`.

## Contributing
See `.github/` directory.