Struct Scanner 

pub struct Scanner { /* private fields */ }

Dataset Scanner

let dataset = Dataset::open(uri).await.unwrap();
let stream = dataset.scan()
    .project(&["col", "col2.subfield"]).unwrap()
    .limit(10)
    .into_stream();
stream
  .map(|batch| batch.num_rows())
  .buffered(16)
  .sum()

Implementations

impl Scanner

pub fn new(dataset: Arc<Dataset>) -> Self

pub fn blob_handling(&mut self, blob_handling: BlobHandling) -> &mut Self

pub fn from_fragment(dataset: Arc<Dataset>, fragment: Fragment) -> Self

pub fn with_fragments(&mut self, fragments: Vec<Fragment>) -> &mut Self

Set which fragments should be scanned.

If scan_in_order is set to true, the fragments will be scanned in the order of the vector.

pub fn empty_project(&mut self) -> Result<&mut Self>

Empty Projection (useful for count queries)

The row_address will be scanned (no I/O required) but not included in the output

pub fn project<T: AsRef<str>>(&mut self, columns: &[T]) -> Result<&mut Self>

Projection.

Only select the specified columns. If not specified, all columns will be scanned.

pub fn project_with_transform( &mut self, columns: &[(impl AsRef<str>, impl AsRef<str>)], ) -> Result<&mut Self>

Projection with transform

Only select the specified columns with the given transform.

pub fn prefilter(&mut self, should_prefilter: bool) -> &mut Self

Should the filter run before the vector index is applied

If true then the filter will be applied before the vector index. This means the results will be accurate but the overall query may be more expensive.

If false then the filter will be applied to the nearest results. This means you may get back fewer results than you ask for (or none at all) if the closest results do not match the filter.

pub fn scan_stats_callback( &mut self, callback: ExecutionStatsCallback, ) -> &mut Self

Set the callback to be called after the scan with summary statistics

pub fn materialization_style( &mut self, style: MaterializationStyle, ) -> &mut Self

Set the materialization style for the scan

This controls when columns are fetched from storage. The default should work well for most cases.

If you know (in advance) a query will return relatively few results (less than 0.1% of the rows) then you may want to experiment with applying late materialization to more (or all) columns.

If you know a query is going to return many rows then you may want to experiment with applying early materialization to more (or all) columns.

pub fn filter(&mut self, filter: &str) -> Result<&mut Self>

Apply filters

The filters can be presented as the string, as in WHERE clause in SQL.

let dataset = Dataset::open(uri).await.unwrap();
let stream = dataset.scan()
    .project(&["col", "col2.subfield"]).unwrap()
    .filter("a > 10 AND b < 200").unwrap()
    .limit(10)
    .into_stream();

Once the filter is applied, Lance will create an optimized I/O plan for filtering.

pub fn filter_query(&mut self, filter: QueryFilter) -> Result<&mut Self>

Apply fts/vector query as filter.

• Vector query filter can only be applied to full text search.
• Fts query filter can only be applied to vector search.
• Query filter couldn’t be applied to normal query.

let dataset = Dataset::open(uri).await.unwrap();
let query_vector = Float32Array::from(vec![300f32, 300f32, 300f32, 300f32]);
let stream = dataset.scan()
    .nearest("vector", &query_vector, 5)
    .project(&["col", "col2.subfield"]).unwrap()
    .query_filter(QueryFilter::Fts(FullTextSearchQuery::new(
      "hello".to_string(),
    ))).unwrap()
    .limit(10)
    .into_stream();

pub fn full_text_search( &mut self, query: FullTextSearchQuery, ) -> Result<&mut Self>

Filter by full text search The column must be a string column. The query is a string to search for. The search is case-insensitive, BM25 scoring is used.

let dataset = Dataset::open(uri).await.unwrap();
let stream = dataset.scan()
   .project(&["col", "col2.subfield"]).unwrap()
   .full_text_search("col", "query").unwrap()
   .limit(10)
   .into_stream();

pub fn filter_substrait(&mut self, filter: &[u8]) -> Result<&mut Self>

Set a filter using a Substrait ExtendedExpression message

The message must contain exactly one expression and that expression must be a scalar expression whose return type is boolean.

pub fn filter_expr(&mut self, filter: Expr) -> &mut Self

pub fn aggregate(&mut self, aggregate: AggregateExpr) -> Result<&mut Self>

Set aggregation.

The aggregate expression is parsed immediately using the dataset schema. For Substrait aggregates, this converts them to DataFusion expressions.

pub fn batch_size(&mut self, batch_size: usize) -> &mut Self

Set the batch size.

pub fn include_deleted_rows(&mut self) -> &mut Self

Include deleted rows

These are rows that have been deleted from the dataset but are still present in the underlying storage. These rows will have the _rowid column set to NULL. The other columns (include _rowaddr) will be set to their deleted values.

This can be useful for generating aligned fragments or debugging

Note: when entire fragments are deleted, the scanner will not emit any rows for that fragment since the fragment is no longer present in the dataset.

pub fn io_buffer_size(&mut self, size: u64) -> &mut Self

Set the I/O buffer size

This is the amount of RAM that will be reserved for holding I/O received from storage before it is processed. This is used to control the amount of memory used by the scanner. If the buffer is full then the scanner will block until the buffer is processed.

Generally this should scale with the number of concurrent I/O threads. The default is 2GiB which comfortably provides enough space for somewhere between 32 and 256 concurrent I/O threads.

This value is not a hard cap on the amount of RAM the scanner will use. Some space is used for the compute (which can be controlled by the batch size) and Lance does not keep track of memory after it is returned to the user.

Currently, if there is a single batch of data which is larger than the io buffer size then the scanner will deadlock. This is a known issue and will be fixed in a future release.

pub fn batch_readahead(&mut self, nbatches: usize) -> &mut Self

Set the prefetch size. Ignored in v2 and newer format

pub fn fragment_readahead(&mut self, nfragments: usize) -> &mut Self

Set the fragment readahead.

This is only used if scan_in_order is set to false.

pub fn scan_in_order(&mut self, ordered: bool) -> &mut Self

Set whether to read data in order (default: true)

A scan will always read from the disk concurrently. If this property is true then a ready batch (a batch that has been read from disk) will only be returned if it is the next batch in the sequence. Otherwise, the batch will be held until the stream catches up. This means the sequence is returned in order but there may be slightly less parallelism.

If this is false, then batches will be returned as soon as they are available, potentially increasing throughput slightly

If an ordering is defined (using Self::order_by) then the scan will always scan in parallel and any value set here will be ignored.

pub fn use_scalar_index(&mut self, use_scalar_index: bool) -> &mut Self

Set whether to use scalar index.

By default, scalar indices will be used to optimize a query if available. However, in some corner cases, scalar indices may not be the best choice. This option allows users to disable scalar indices for a query.

pub fn strict_batch_size(&mut self, strict_batch_size: bool) -> &mut Self

Set whether to use strict batch size.

If this is true then output batches (except the last batch) will have exactly batch_size rows. By default, this is False and output batches are allowed to have fewer than batch_size rows Setting this to True will require us to merge batches, incurring a data copy, for a minor performance penalty.

pub fn limit( &mut self, limit: Option<i64>, offset: Option<i64>, ) -> Result<&mut Self>

Set limit and offset.

If offset is set, the first offset rows will be skipped. If limit is set, only the provided number of rows will be returned. These can be set independently. For example, setting offset to 10 and limit to None will skip the first 10 rows and return the rest of the rows in the dataset.

pub fn nearest( &mut self, column: &str, q: &dyn Array, k: usize, ) -> Result<&mut Self>

Find k-nearest neighbor within the vector column. the query can be a Float16Array, Float32Array, Float64Array, UInt8Array, or a ListArray/FixedSizeListArray of the above types.

pub fn distance_range( &mut self, lower_bound: Option<f32>, upper_bound: Option<f32>, ) -> &mut Self

Set the distance thresholds for the nearest neighbor search.

pub fn nprobes(&mut self, n: usize) -> &mut Self

Configures how many partititions will be searched in the vector index.

This method is a convenience method that sets both Self::minimum_nprobes and Self::maximum_nprobes to the same value.

pub fn nprobs(&mut self, n: usize) -> &mut Self

👎Deprecated:

Use nprobes instead

Configures how many partititions will be searched in the vector index.

This method is a convenience method that sets both Self::minimum_nprobes and Self::maximum_nprobes to the same value.

pub fn minimum_nprobes(&mut self, n: usize) -> &mut Self

Configures the minimum number of partitions to search in the vector index.

If we have found k matching results after searching this many partitions then the search will stop. Increasing this number can increase recall but will increase latency on all queries.

The default value is 1.

pub fn maximum_nprobes(&mut self, n: usize) -> &mut Self

Configures the maximum number of partitions to search in the vector index.

These partitions will only be searched if we have not found k results after searching the minimum number of partitions. Setting this to None (the default) will search all partitions if needed.

This setting only takes effect when a prefilter is in place. In that case we can spend more effort to try and find results when the filter is highly selective.

If there is no prefilter, or the results are not highly selective, this value will have no effect.

pub fn ef(&mut self, ef: usize) -> &mut Self

pub fn fast_search(&mut self) -> &mut Self

Only search the data being indexed.

Default value is false.

This is essentially a weak consistency search, only on the indexed data.

pub fn refine(&mut self, factor: u32) -> &mut Self

Apply a refine step to the vector search.

A refine improves query accuracy but also makes search slower, by reading extra elements and using the original vector values to re-rank the distances.

• factor - the factor of extra elements to read. For example, if factor is 2, then the search will read 2x more elements than the requested k before performing the re-ranking. Note: even if the factor is 1, the results will still be re-ranked without fetching additional elements.

pub fn distance_metric(&mut self, metric_type: MetricType) -> &mut Self

Change the distance MetricType, i.e, L2 or Cosine distance.

pub fn order_by( &mut self, ordering: Option<Vec<ColumnOrdering>>, ) -> Result<&mut Self>

Sort the results of the scan by one or more columns

If Some, then the resulting stream will be sorted according to the given ordering. This may increase the latency of the first result since all data must be read before the first batch can be returned.

pub fn use_index(&mut self, use_index: bool) -> &mut Self

Set whether to use the index if available

pub fn with_row_id(&mut self) -> &mut Self

Instruct the scanner to return the _rowid meta column from the dataset.

pub fn with_row_address(&mut self) -> &mut Self

Instruct the scanner to return the _rowaddr meta column from the dataset.

pub fn disable_scoring_autoprojection(&mut self) -> &mut Self

Instruct the scanner to disable automatic projection of scoring columns

In the future, this will be the default behavior. This method is useful for opting in to the new behavior early to avoid breaking changes (and a warning message)

Once the default switches, the old autoprojection behavior will be removed.

The autoprojection behavior (current default) includes the _score or _distance column even if a projection is manually specified with [project] or [project_with_transform].

The new behavior will only include the _score or _distance column if no projection is specified or if the user explicitly includes the _score or _distance column in the projection.

pub fn with_file_reader_options( &mut self, options: FileReaderOptions, ) -> &mut Self

Set the file reader options to use when reading data files.

pub fn use_stats(&mut self, use_stats: bool) -> &mut Self

Set whether to use statistics to optimize the scan (default: true)

This is used for debugging or benchmarking purposes.

pub async fn schema(&self) -> Result<SchemaRef>

The Arrow schema of the output, including projections and vector / _distance

pub fn get_expr_filter(&self) -> Result<Option<Expr>>

Fetches the currently set expr filter

Note that this forces the filter to be evaluated and the result will depend on the current state of the scanner (e.g. if with_row_id has been called then _rowid will be available for filtering but not otherwise) and so you may want to call this after setting all other options.

pub fn try_into_stream(&self) -> BoxFuture<'_, Result<DatasetRecordBatchStream>>

Create a stream from the Scanner.

pub async fn try_into_batch(&self) -> Result<RecordBatch>

pub fn count_rows(&self) -> BoxFuture<'_, Result<u64>>

Scan and return the number of matching rows

Note: calling Dataset::count_rows can be more efficient than calling this method especially if there is no filter.

pub fn create_aggregate_plan( &self, ) -> BoxFuture<'_, Result<Arc<dyn ExecutionPlan>>>

👎Deprecated:

Use create_plan() instead, which now applies aggregate automatically

Create an execution plan with aggregation.

Requires aggregate() to be called first.

pub async fn create_plan(&self) -> Result<Arc<dyn ExecutionPlan>>

Create ExecutionPlan for Scan.

An ExecutionPlan is a graph of operators that can be executed.

The following plans are supported:

• Plain scan without filter or limits.

Scan(projections)

• Scan with filter and/or limits.

Scan(filtered_cols) -> Filter(expr)
   -> (*LimitExec(limit, offset))
   -> Take(remaining_cols) -> Projection()

• Use KNN Index (with filter and/or limits)

KNNIndex() -> Take(vector) -> FlatRefine()
    -> Take(filtered_cols) -> Filter(expr)
    -> (*LimitExec(limit, offset))
    -> Take(remaining_cols) -> Projection()

• Use KNN flat (brute force) with filter and/or limits

Scan(vector) -> FlatKNN()
    -> Take(filtered_cols) -> Filter(expr)
    -> (*LimitExec(limit, offset))
    -> Take(remaining_cols) -> Projection()

In general, a plan has 5 stages:

1. Source (from dataset Scan or from index, may include prefilter)
2. Filter
3. Sort
4. Limit / Offset
5. Take remaining columns / Projection

pub async fn analyze_plan(&self) -> Result<String>

pub async fn explain_plan(&self, verbose: bool) -> Result<String>

Trait Implementations

impl Clone for Scanner

fn clone(&self) -> Scanner

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.