Struct LanceStorageGraph 

pub struct LanceStorageGraph { /* private fields */ }

Lance-based storage backend for ArrowSpace graph embeddings.

Stores dense and sparse matrices as Lance datasets using a columnar format (row, col, value for sparse; col_* for dense) schema for efficient random and columnar access.

Metadata must be seeded before any save_* call so the storage directory is initialized; the example below is executed as a doc-test on every cargo test run so it cannot go stale.

Examples

use genegraph_storage::lance_storage_graph::LanceStorageGraph;
use genegraph_storage::metadata::GeneMetadata;
use genegraph_storage::traits::backend::StorageBackend;
use genegraph_storage::traits::metadata::Metadata;
use smartcore::linalg::basic::arrays::{Array, Array2};
use smartcore::linalg::basic::matrix::DenseMatrix;

let base = std::env::temp_dir().join(format!("genegraph_doc_{}", std::process::id()));
let storage = LanceStorageGraph::new(
    base.to_string_lossy().to_string(),
    "doc_example".to_string(),
)
.expect("valid instance name");

// some 2D data
let dense: Vec<Vec<f64>> = vec![vec![0.1, 0.4], vec![0.5, 0.2], vec![0.03, 0.8]];
let (nitems, nfeatures) = (dense.len(), dense[0].len());
let data = DenseMatrix::<f64>::from_iterator(
    dense.iter().flatten().copied(),
    nitems,
    nfeatures,
    0,
);

// seed metadata FIRST to initialize the storage directory
let md = GeneMetadata::seed_metadata("doc_example", nitems, nfeatures, &storage)
    .await
    .unwrap();
let md_path = storage.save_metadata(&md).await.unwrap();

// your data is saved in an efficient Lance format
storage.save_dense("my_dataset", &data, &md_path).await.unwrap();

// Loading back
let loaded = storage.load_dense("my_dataset").await.unwrap();
assert_eq!(loaded.shape(), (nitems, nfeatures));

Implementations

impl LanceStorageGraph

pub fn new(base: String, name: String) -> StorageResult<Self>

Creates a new Lance storage backend.

This is used for on-the-fly creation. For proper setup use Genefold<...>::seed.

Arguments

• base - Base directory path for all storage files
• name - Name prefix for this storage instance

The name must not carry the reserved generation suffix __g<n> (#117-3): only Self::scoped_generation mints generation-qualified names, otherwise generation discovery and sweep would treat the instance as a generation of another logical dataset.

pub async fn spawn( base_path: String, ) -> Result<(Self, GeneMetadata), StorageError>

Spawn a LanceStorage from an existing seeded directory (with metadata.json)

pub fn scoped_generation(&self, generation: u64) -> Self

A handle over generation gen of this logical dataset: artifact paths resolve as {logical}__g{gen}_{key}.lance and the metadata path as {logical}__g{gen}_metadata.json — the per-generation commit pointer (#93, RFC #81-P5). The logical identity stays recoverable via crate::generations::logical_name.

Trait Implementations

impl Clone for LanceStorageGraph

fn clone(&self) -> LanceStorageGraph

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl ClusteringArtifacts for LanceStorageGraph

async fn save_centroid_map( &self, map: &[usize], md_path: &Path, ) -> StorageResult<()>

Save centroid_map (item-to-centroid assignments)

async fn load_centroid_map(&self) -> StorageResult<Vec<usize>>

Load centroid_map

async fn save_subcentroid_lambdas( &self, lambdas: &[f64], md_path: &Path, ) -> StorageResult<()>

Save subcentroid_lambdas (tau values for subcentroids)

async fn load_subcentroid_lambdas(&self) -> StorageResult<Vec<f64>>

Load subcentroid_lambdas

async fn save_subcentroids( &self, subcentroids: &DenseMatrix<f64>, md_path: &Path, ) -> StorageResult<()>

Save subcentroids (dense matrix)

async fn load_subcentroids(&self) -> StorageResult<Vec<Vec<f64>>>

Load subcentroids as Vec<Vec>

async fn save_item_norms( &self, item_norms: &[f64], md_path: &Path, ) -> StorageResult<()>

Save item norms vector

async fn load_item_norms(&self) -> StorageResult<Vec<f64>>

Load item norms vector

async fn save_cluster_assignments( &self, assignments: &[Option<usize>], md_path: &Path, ) -> StorageResult<()>

Save cluster assignments (Vec<Option>)

async fn load_cluster_assignments(&self) -> StorageResult<Vec<Option<usize>>>

Load cluster assignments

impl Debug for LanceStorageGraph

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl LanceStorage for LanceStorageGraph

async fn write_lance_batch_async( &self, uri: String, batch: RecordBatch, ) -> StorageResult<()>

Async helper: write a RecordBatch to a Lance dataset.

async fn read_lance_all_batches_async( &self, uri: String, ) -> StorageResult<RecordBatch>

Async helper: read and concatenate all RecordBatches from a Lance dataset.

async fn save_primitive_column<T: ArrowPrimitiveType>( &self, key: &str, field_name: &str, values: Vec<T::Native>, md_path: &Path, ) -> StorageResult<()>

where Self: StorageBackend,

Writes a single-column primitive vector as a Lance dataset named <name>_<key>.lance and registers it in the metadata files map under key with filetype “vector” and shape (<len>, 1).

impl StorageBackend for LanceStorageGraph

fn basepath_to_uri(&self) -> StorageResult<String>

Converts the base path for the store to a file:// URI for Lance.

async fn save_dense( &self, key: &str, matrix: &DenseMatrix<f64>, md_path: &Path, ) -> StorageResult<()>

Save dense matrix using Lance-optimized vector format.

Each row of the matrix becomes a FixedSizeList entry for efficient vector operations. This format is optimized for vector search and enables Lance’s full-zip encoding.

Arguments

• filename - any name
• matrix - Dense matrix to save (N rows × F cols)
• md_path - Metadata file path for validation

async fn load_dense(&self, key: &str) -> StorageResult<DenseMatrix<f64>>

Load dense matrix from Lance-optimized vector format.

Reads FixedSizeList vectors and reconstructs a column-major DenseMatrix.

Arguments

• filename - any name previously assigned

Returns

Column-major DenseMatrix matching smartcore conventions

async fn load_dense_from_file( &self, path: &Path, ) -> StorageResult<DenseMatrix<f64>>

Load initial data using columnar format from a file path.

Async test helper that avoids any internal blocking runtimes. Format dispatch only (#117-4): each branch lives in its own helper.

async fn save_dense_to_file( data: &DenseMatrix<f64>, path: &Path, ) -> StorageResult<()>

Save dense matrix to file in columnar format (col_0, col_1, …, col_N)

Async test helper that avoids any internal blocking runtimes.

fn get_base(&self) -> String

Base directory of the instance

fn get_name(&self) -> String

Name of the instance

fn base_path(&self) -> PathBuf

Returns the base directory path.

fn metadata_path(&self) -> PathBuf

Returns the metadata path.

fn file_path(&self, key: &str) -> PathBuf

Compute the full Lance/parquet file path for a logical filetype.

async fn save_sparse( &self, key: &str, matrix: &CsMat<f64>, md_path: &Path, ) -> StorageResult<()>

Saves a sparse matrix. Requires metadata to exist.

async fn load_sparse(&self, key: &str) -> StorageResult<CsMat<f64>>

Loads a sparse matrix from storage.

async fn save_lambdas( &self, lambdas: &[f64], md_path: &Path, ) -> StorageResult<()>

Saves lambda eigenvalues. Requires metadata to exist.

async fn load_lambdas(&self) -> StorageResult<Vec<f64>>

Loads lambda eigenvalues from storage.

async fn save_vector( &self, key: &str, vector: &[f64], md_path: &Path, ) -> StorageResult<()>

save a generic f64 sequence

async fn save_index( &self, key: &str, vector: &[usize], md_path: &Path, ) -> StorageResult<()>

Save vectors that are not lambdas but indices.

async fn load_vector(&self, filename: &str) -> StorageResult<Vec<f64>>

async fn load_index(&self, filename: &str) -> StorageResult<Vec<usize>>

Load index or generic usize vector from storage.

async fn save_vectors_with( &self, name: &str, batch: &RecordBatch, properties: &BTreeMap<String, String>, md_path: &Path, ) -> StorageResult<()>

Saves a vector-space collection with user properties (e.g. the graph linkage of RFC #81-P4: properties.graph = <name>).

async fn load_vectors(&self, name: &str) -> StorageResult<RecordBatch>

Loads a vector-space collection back as a RecordBatch with its original schema (including collection metadata).

async fn save_graph_with( &self, name: &str, edges: &[GraphEdge], options: &GraphWriteOptions, md_path: &Path, ) -> StorageResult<()>

Saves a graph collection with explicit options (node-id and weight widths, node count, user properties for vector-space linkage, RFC #81-P4).

async fn load_graph(&self, name: &str) -> StorageResult<StoredGraph>

Loads a graph collection back as an edge list; convert to CSR at the API boundary with StoredGraph::to_csr.

async fn save_vectors_to_path( &self, path: &Path, batch: &RecordBatch, properties: &BTreeMap<String, String>, ) -> StorageResult<()>

Saves a vector-space collection dataset at an explicit path without touching the metadata registry (#106): the batch is validated, stamped with dataset-level collection metadata (including user properties) and written; parent directories are created as needed. Registry ownership stays with the caller — the counterpart of Self::save_vectors_with for consumers that run their own metadata registry (e.g. an ArrowSpaceMetadata commit pointer at the instance metadata path). No GeneMetadata read or write occurs.

async fn load_vectors_from_path( &self, path: &Path, ) -> StorageResult<RecordBatch>

Loads a vector-space collection dataset from an explicit path (registry-free): the path-based counterpart of Self::load_vectors.

async fn save_graph_to_path( &self, path: &Path, edges: &[GraphEdge], options: &GraphWriteOptions, ) -> StorageResult<()>

Saves a graph collection dataset at an explicit path without touching the metadata registry (#106); see Self::save_graph_with for the options and Self::save_vectors_to_path for the registry-free contract.

async fn load_graph_from_path(&self, path: &Path) -> StorageResult<StoredGraph>

Loads a graph collection dataset from an explicit path (registry-free): the path-based counterpart of Self::load_graph.

async fn load_graph_from_path_with_options( &self, path: &Path, options: &GraphReadOptions, ) -> StorageResult<StoredGraph>

Loads a graph collection from an explicit path with read options (#108): strict mode rejects datasets without a num_nodes stamp (and, optionally, validates column names) instead of silently resizing to max_id + 1. The tolerant default is the compat shim.

async fn load_scalars(&self, name: &str) -> StorageResult<Vec<f64>>

Loads a scalar collection — a single Float64|Float32 column, e.g. lambdas or norms — as Vec<f64> (#106); f32 values are upcast losslessly. The stamped dataset kind must be vector-space (missing or mismatched kinds are rejected: physical shape determines decodability, the logical kind determines semantic validity), making every kind=vector-space collection uniformly loadable instead of requiring the legacy fixed-key readers.

async fn save_scalars_to_path( &self, path: &Path, values: &[f64], ) -> StorageResult<()>

Saves a scalar collection — a single non-nullable Float64 column (e.g. lambdas or norms) — at an explicit path without touching the metadata registry (#107): the dataset is stamped kind=vector-space and parent directories are created as needed. Registry ownership stays with the caller; no GeneMetadata read or write occurs. The counterpart of Self::save_vectors_to_path for scalar-only collections (which save_vectors_to_path cannot write, as its schema validation requires a FixedSizeList column).

async fn load_scalars_from_path(&self, path: &Path) -> StorageResult<Vec<f64>>

Loads a scalar collection from an explicit path (registry-free): the path-based counterpart of Self::load_scalars, with the same kind gate. load_scalars delegates to this.

async fn collection_schema_from_path( &self, path: &Path, ) -> StorageResult<Schema>

Reads the dataset schema (fields + KV metadata) of a collection at an explicit path without decoding any column buffers (#109): the backend surface of crate::lancefmt::read_schema, so consumers doing kind/stamp gating don’t pay a full columnar scan.

fn exists(path: &str) -> (bool, Option<PathBuf>)

Returns true and the path to the metadata file if metadata file exists and is valid, false otherwise. This is used to avoid overwriting existing indexes.

fn path_to_uri(path: &Path) -> StorageResult<String>

Converts a full file path to a file:// URI for Lance.

fn validate_initialized(&self, md_path: &Path) -> StorageResult<()>

Validates that the storage directory is properly initialized with metadata.

fn to_dense_record_batch( &self, matrix: &DenseMatrix<f64>, ) -> Result<RecordBatch, StorageError>

Converts a dense matrix to a RecordBatch in vector format (Lance-optimized). Each row of the matrix becomes a single FixedSizeList entry.

fn from_dense_record_batch( &self, batch: &RecordBatch, ) -> Result<DenseMatrix<f64>, StorageError>

Reconstructs a dense matrix from a RecordBatch in vector format.

fn to_sparse_record_batch(&self, m: &CsMat<f64>) -> StorageResult<RecordBatch>

Converts a sparse CSR matrix to a RecordBatch in columnar format.

fn from_sparse_record_batch( &self, batch: RecordBatch, expected_rows: usize, expected_cols: usize, ) -> StorageResult<CsMat<f64>>

Reconstructs a sparse CSR matrix from a RecordBatch in columnar format.

async fn save_metadata(&self, metadata: &GeneMetadata) -> StorageResult<PathBuf>

Initializes storage by saving metadata. Must be called first.

async fn load_metadata(&self) -> StorageResult<GeneMetadata>

Loads metadata from storage.

async fn save_vectors( &self, name: &str, batch: &RecordBatch, md_path: &Path, ) -> StorageResult<()>

Saves a schema-driven vector-space collection (RFC #81-P2) without user properties; see Self::save_vectors_with.

async fn save_graph( &self, name: &str, edges: &[GraphEdge], md_path: &Path, ) -> StorageResult<()>

Saves an edge-list graph collection (RFC #81-P3) with u32 node ids (the default width) and f64 weights (the default width, mirroring the sparse-matrix value column). Ids above u32::MAX surface StorageError::Overflow instead of being truncated. Weights are persisted faithfully (normalization transforms belong to the producer); the opt-in GraphWriteOptions::weight_range asserts compliance with a declared closed interval, and weights that cannot be stored exactly at an explicitly declared crate::graph::WeightType::F32 width are rejected instead of being silently narrowed.

async fn load_graph_from_path_strict( &self, path: &Path, ) -> StorageResult<StoredGraph>

Strict variant of Self::load_graph_from_path (#108): rejects datasets without a num_nodes stamp with a typed StorageError::Invalid and validates the src/dst/weight column names, so pre-collections triplet artifacts are positively identified rather than positionally coerced.

async fn collection_schema(&self, name: &str) -> StorageResult<Schema>

Reads the dataset schema of a named collection (registry-free, path-resolved): the name-based counterpart of Self::collection_schema_from_path.

async fn verify_collection_from_path( &self, path: &Path, expected: &DatasetFacts, ) -> StorageResult<()>

Verifies the stamped dataset schema of a collection at path against expected (#115): reads the schema footer via crate::lancefmt::read_schema — no column buffers are decoded — and cross-checks the facts (kind, rows/cols, num_nodes, weight width). Mismatches surface as typed errors naming the fact.

async fn verify_collection( &self, name: &str, expected: &DatasetFacts, ) -> StorageResult<()>

Name-based counterpart of Self::verify_collection_from_path, resolving through Self::file_path.