genegraph-storage 0.20.0

vector database: base Lance storage
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

//! In-house implementation of the Lance v2.1 format subset.
//!
//! Canonical plan: [#75](https://github.com/tuned-org-uk/genegraph-storage/issues/75).
//! Framing/motivation: [#74](https://github.com/tuned-org-uk/genegraph-storage/issues/74).
//!
//! # Spec pin
//!
//! The format targets **Lance v2.1**; the protobuf definitions are vendored
//! verbatim under `src/lancefmt/protos/` (with provenance and license notes)
//! from the `lance` crate v11.0.0. Files written by the official crate that
//! use encodings outside our minimal set are rejected with
//! `StorageError::UnsupportedFormat`, never guessed at.
//!
//! # Supported subset
//!
//! - Non-nullable top-level columns of `Float64`, `UInt32` or
//!   `FixedSizeList<Float64>` (dense vectors)
//! - Single fragment / single data file, MiniBlock pages, Flat or
//!   InlineBitpacking value compression, no repetition/definition levels
//!
//! # Milestones
//!
//! - M0: conformance harness — vendored spec, golden fixtures generated by
//!   the official crate, round-trip tests.
//! - M1 (this commit): writer — single fragment, Flat encodings, manifest v1.
//! - M2 (this commit): reader — full scan.
//! - M2.5: parquet interop promotion.
//! - M-C1: catalog contract (`Catalog` trait + `LocalRegistry`).
//! - M3: take/projection (conditional on consumer survey).
//! - M4: overwrite via manifest versioning.
//! - M5: swap internals behind `StorageBackend`; drop runtime `lance`.

pub mod pb;
mod reader;
mod schema;
mod writer;

pub use reader::scan_all;
pub use writer::write_dataset;