Crate lance_nd 

N-dimensional Arrow arrays in Lance.

A table is a Lance dataset whose schema is the nd-arrow encoding: one nd_arrow.array column per array name and an optional __global_attrs__ column. Each row is one NdRecordBatch, a complete one or a chunk of a larger one.

• open() and open_with(): open a table and check its format version. StorageOptions hold the credentials of S3, Google Cloud Storage or Azure.
• write() and append(): write a stream of nd record batches.
• NdWriter: write in several calls and commit once.
• scan(): read a stream of nd record batches, with a projection and filters. prune_filter() shows how a filter on values uses the stats.
• delete(), compact() and cleanup_old_versions(): delete rows and reclaim space.
• versions(), checkout() and restore(): read and restore earlier versions.
• widen(): change the value type of a variable.
• ColumnOptions: Lance encoding options, like the compression, per column.
• encode_rows() and RowDecoder: the row encoding, for record batches that you write or read yourself.

See FORMAT.md for the table layout.

Re-exports

pub use lance::deps::datafusion;

Structs

CleanupStats

What a cleanup_old_versions removed.

ColumnOptions

Lance encoding options of one column. None leaves the choice to Lance.

CompactOptions

Options for compact.

CompactionStats

What a compact changed.

NdScanner

A scan that yields one nd record batch per row, in table order.

NdWriter

A writer that makes all its rows visible in one commit.

RowDecoder

Decodes the rows of one record batch of a table, one row at a time.

TableVersion

One version of a table.

WriteOptions

Options for write() and append().

Enums

ByteStreamSplit

Byte stream split of floating point values before compression.

Casting

How a write treats a value or attribute type that differs from the table.

Compression

The general compression of the values of a column.

Error

Errors raised by lance-nd.

StructuralEncoding

The page layout of a column.

WriteMode

How a write treats the table at its uri.

Constants

FORMAT_VERSION

The format version that this release reads and writes.

STATS_PREFIX

The name prefix of the stats columns. Each variable name that starts with it is reserved.

Functions

append

Append a stream of rows to the table of dataset in one commit. Each nd record batch is one row. A failed row commits nothing.

check_format

Fails when dataset is not a lance-nd table of FORMAT_VERSION.

checkout

The table at version, to read. A write to it fails, because it is not the latest version.

cleanup_old_versions

Remove the files of the table versions older than older_than. The current version always stays.

column_defaults

The column defaults of the table. New columns get them.

compact

Merge small fragments, split large fragments and remove deleted rows. Keeps the row order.

delete

Delete the rows that match predicate, a Lance SQL filter.

delete_expr

Delete the rows that match expr, a DataFusion filter on the Lance table columns.

encode_rows

Encode rows as one record batch, one row per nd record batch.

has_stats

true when the schema marks a table with stats.

open

Open the lance-nd table at uri. Fails when no table is there, or when the table has another format version.

open_with

Open the lance-nd table at uri with Lance storage options, for example the credentials of an object store. See crate::StorageOptions.

prune_filter

A filter for a scan of table with the rows that can match expr. None means that every row is kept. crate::NdScanner::filter_expr uses it.

restore

Make the content of version the latest version of the table, in a new commit. The versions after it stay in the history.

scan

A scan over the rows of the table.

set_column_defaults

Replace the column defaults of the table. Existing columns keep their encoding.

stats_column

The name of the stats column of variable.

stats_variable

The variable of the stats column column, or None for another name.

variables

The variables of the table, as logical fields typed by their values.

versions

The versions of the table that still have their files, oldest first.

widen

Change the value type of variable to to, for example Float32 to Float64. The stats of the variable get the new type too.

write

Write a stream of rows to the table at uri in one commit. Each nd record batch is one row.

Type Aliases

Result

A result with Error as the default error.

StorageOptions

Lance storage options, by key, for tables on object stores.