Crate lance

Lance Columnar Data Format

Lance columnar data format is an alternative to Parquet. It provides 100x faster for random access, automatic versioning, optimized for computer vision, bioinformatics, spatial and ML data. Apache Arrow and DuckDB compatible.

Create a Dataset

use lance::{dataset::WriteParams, Dataset};

let schema = Arc::new(Schema::new(vec![Field::new("test", DataType::Int64, false)]));
let batches = vec![RecordBatch::new_empty(schema.clone())];
let reader = RecordBatchIterator::new(
    batches.into_iter().map(Ok), schema
);

let write_params = WriteParams::default();
Dataset::write(reader, &uri, Some(write_params)).await.unwrap();

Scan a Dataset

use futures::StreamExt;
use lance::Dataset;

let dataset = Dataset::open(&path).await.unwrap();
let mut scanner = dataset.scan();
let batches: Vec<RecordBatch> = scanner
    .try_into_stream()
    .await
    .unwrap()
    .map(|b| b.unwrap())
    .collect::<Vec<RecordBatch>>()
    .await;

Re-exports

• pub use dataset::Dataset;

Modules

• arrow
  Extend Arrow Functionality
• datafusion
  Extends DataFusion
• dataset
  Lance Dataset
• datatypes
  Lance data types, Schema and Field
• error
• index
  Secondary Index
• io
  I/O utilities.
• session
• table
• utils
  Various utilities

Structs

• DIST_FIELD
• RESULT_SCHEMA
  Row ID field. This is nullable because its validity bitmap is sometimes used as a selection vector.

Enums

• Error

Functions

• open_dataset
  Creates and loads a Dataset from the given path. Infers the storage backend to use from the scheme in the given table path.

Type Aliases

• Result