lance-nd 0.1.0

lance-nd stores N-dimensional array data in a Lance table. Each table row is one nd record batch of nd-arrow-array: a set of named N-dimensional arrays with their dimensions and attributes, such as one xarray dataset.

• The Lance table schema is the nd-arrow encoding, so any Arrow or Lance reader can read the data.
• Rows can have different variables. New variables and attributes extend the table schema.
• Optional per-row stats let a scan skip rows on value filters.
• Tables live on local disk, S3, Google Cloud Storage or Azure.

Install

[dependencies]
lance-nd = "0.1"
nd-arrow-array = "0.5"

Turn on the cargo feature aws, gcp or azure for an object store. Lance needs protoc to build.

Rust

use futures::{StreamExt, stream};
use lance_nd::{WriteOptions, scan, write};

// `batches` is an iterator of nd_arrow_array::NdRecordBatch.
let rows = stream::iter(batches.map(Ok));
let table = write("data/arrays.lance", rows, &WriteOptions::default()).await?;

let mut rows = scan(&table)
    .project(&["temperature"])
    .filter("__global_attrs__.id = 'a'")
    .try_into_stream()
    .await?;
while let Some(batch) = rows.next().await {
    println!("{:?}", batch?.target());
}

Python

import xarray as xr
import lance_nd

with lance_nd.create("data/arrays.lance", stats=True) as writer:
    writer.write(xr.open_dataset("obs.nc"))

table = lance_nd.open("data/arrays.lance")
for batch in table.scan(project=["sst"], prune="sst.values > 300"):
    ds = batch.to_xarray()

See python/README.md to install the package.

Documentation

The wiki explains each part in depth:

• Table layout: the data model, columns and schema evolution.
• Writing: NdWriter, errors, fragments, casting and column encoding.
• Reading and filters: stats, pruning and filters.
• Object stores: S3, Google Cloud Storage and Azure.
• Delete, compact and versions.
• Python: the full Python API and the encoding rules.

FORMAT.md is the specification of the lance-nd table.

Status

lance-nd is experimental. It has no indexes. A table is a lance::Dataset, so a new major version of Lance is a breaking change of lance-nd.