Struct NdScanner 

pub struct NdScanner<'a> { /* private fields */ }

A scan that yields one nd record batch per row, in table order.

Lance reads batches of rows, limited by rows and by bytes. Several batches decode to nd record batches at the same time.

Implementations

impl NdScanner<'_>

pub fn project(self, variables: &[&str]) -> Self

Read only these variables. A row keeps its global attributes. A row without any of these variables gives a batch without columns, so an empty list reads only the global attributes.

Each name must be a whole variable. A nested field, such as sst.attrs.units, gives Error::InvalidArgument in the scan.

pub fn filter(self, filter: impl Into<String>) -> Self

Read only the rows that match a Lance SQL filter.

A scan cannot have both this filter and a DataFusion filter, see NdScanner::filter_expr and NdScanner::lance_filter.

pub fn filter_expr(self, expr: Expr) -> Self

Skip the rows that surely have no value that matches expr.

expr is a DataFusion filter on the table columns, for example get_field(col("lat"), "values").between(lit(17.5), lit(25.4)). A part on <variable>.values uses the stats of the table, see crate::prune_filter. A kept row comes back whole, also with values outside the filter. A table without stats keeps all rows for a value part. Each other part, for example on attributes, filters the rows exactly.

pub fn lance_filter(self, expr: Expr) -> Self

Read only the rows that match expr, a DataFusion filter on the Lance table columns.

Lance gets expr with no change, so expr must use the table columns: the nd columns, __global_attrs__ and the stats columns. With NdScanner::filter_expr, a row must match both filters.

pub fn fragments(self, fragments: Vec<Fragment>) -> Self

Read only the rows of these fragments, in the order given.

pub fn limit(self, limit: usize) -> Self

Read at most limit rows.

pub fn batch_size(self, rows: usize) -> Self

The most rows in one Lance batch. The default is 256.

pub fn batch_size_bytes(self, bytes: u64) -> Self

The approximate most bytes in one Lance batch. The default is 32 MiB. A row larger than this is a batch of its own.

pub fn decode_parallelism(self, batches: usize) -> Self

The most Lance batches that decode at the same time. The default is the number of compute CPUs of Lance.

pub fn batch_readahead(self, batches: usize) -> Self

The most Lance batches that Lance decodes ahead of the consumer.

pub async fn explain_plan(&self, verbose: bool) -> Result<String>

The Lance plan of the scan, for example to see which columns the filter read loads.

pub async fn try_into_stream( self, ) -> Result<BoxStream<'static, Result<NdRecordBatch>>>

A stream of the selected rows, in table order.

pub async fn try_collect(self) -> Result<Vec<NdRecordBatch>>

Collect the selected rows.