Struct Table

pub struct Table { /* private fields */ }

A Table is a collection of strong typed Rows.

The type of the each row is defined in Apache Arrow Schema.

Implementations

impl Table

pub fn new(inner: Arc<dyn BaseTable>) -> Self

pub fn base_table(&self) -> &Arc<dyn BaseTable>

pub fn as_native(&self) -> Option<&NativeTable>

Cast as NativeTable, or return None it if is not a NativeTable.

Warning: This function will be removed soon (features exclusive to NativeTable will be added to Table)

pub fn name(&self) -> &str

Get the name of the table.

pub fn dataset(&self) -> Option<&DatasetConsistencyWrapper>

Get the dataset of the table if it is a native table

Returns None otherwise

pub async fn schema(&self) -> Result<SchemaRef>

Get the arrow Schema of the table.

pub async fn count_rows(&self, filter: Option<String>) -> Result<usize>

Count the number of rows in this dataset.

Arguments

• filter if present, only count rows matching the filter

pub fn add<T: IntoArrow>(&self, batches: T) -> AddDataBuilder<T>

Insert new records into this Table

Arguments

• batches data to be added to the Table
• options options to control how data is added

pub fn update(&self) -> UpdateBuilder

Update existing records in the Table

An update operation can be used to adjust existing values. Use the returned builder to specify which columns to update. The new value can be a literal value (e.g. replacing nulls with some default value) or an expression applied to the old value (e.g. incrementing a value)

An optional condition can be specified (e.g. “only update if the old value is 0”)

Note: if your condition is something like “some_id_column == 7” and you are updating many rows (with different ids) then you will get better performance with a single [merge_insert] call instead of repeatedly calilng this method.

pub async fn delete(&self, predicate: &str) -> Result<DeleteResult>

Delete the rows from table that match the predicate.

Arguments

• predicate - The SQL predicate string to filter the rows to be deleted.

Example

let tmpdir = tempfile::tempdir().unwrap();
let db = lancedb::connect(tmpdir.path().to_str().unwrap())
    .execute()
    .await
    .unwrap();
let batches = RecordBatchIterator::new(
    vec![RecordBatch::try_new(
        schema.clone(),
        vec![
            Arc::new(Int32Array::from_iter_values(0..10)),
            Arc::new(
                FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
                    (0..10).map(|_| Some(vec![Some(1.0); 128])),
                    128,
                ),
            ),
        ],
    )
    .unwrap()]
    .into_iter()
    .map(Ok),
    schema.clone(),
);
let tbl = db
    .create_table("delete_test", Box::new(batches))
    .execute()
    .await
    .unwrap();
tbl.delete("id > 5").await.unwrap();

pub fn create_index( &self, columns: &[impl AsRef<str>], index: Index, ) -> IndexBuilder

Create an index on the provided column(s).

Indices are used to speed up searches and are often needed when the size of the table becomes large (the exact size depends on many factors but somewhere between 100K rows and 1M rows is a good rule of thumb)

There are a variety of indices available. They are described more in crate::index::Index. The simplest thing to do is to use index::Index::Auto which will attempt to create the most useful index based on the column type and column statistics. BTree index is created by default for numeric, temporal, and string columns.

Once an index is created it will remain until the data is overwritten (e.g. an add operation with mode overwrite) or the indexed column is dropped.

Indices are not automatically updated with new data. If you add new data to the table then the index will not include the new rows. However, a table search will still consider the unindexed rows. Searches will issue both an indexed search (on the data covered by the index) and a flat search (on the unindexed data) and the results will be combined.

If there is enough unindexed data then the flat search will become slow and the index should be optimized. Optimizing an index will add any unindexed data to the existing index without rerunning the full index creation process. For more details see Table::optimize.

Note: Multi-column (composite) indices are not currently supported. However, they will be supported in the future and the API is designed to be compatible with them.

Examples

use lancedb::index::Index;
let tmpdir = tempfile::tempdir().unwrap();
let db = lancedb::connect(tmpdir.path().to_str().unwrap())
    .execute()
    .await
    .unwrap();
// Create IVF PQ index on the "vector" column by default.
tbl.create_index(&["vector"], Index::Auto)
   .execute()
   .await
   .unwrap();
// Create a BTree index on the "id" column.
tbl.create_index(&["id"], Index::Auto)
    .execute()
    .await
    .unwrap();
// Create a LabelList index on the "tags" column.
tbl.create_index(&["tags"], Index::LabelList(Default::default()))
    .execute()
    .await
    .unwrap();

pub fn create_index_with_timeout( &self, columns: &[impl AsRef<str>], index: Index, wait_timeout: Option<Duration>, ) -> IndexBuilder

See Table::create_index For remote tables, this allows an optional wait_timeout to poll until asynchronous indexing is complete

pub fn merge_insert(&self, on: &[&str]) -> MergeInsertBuilder

Create a builder for a merge insert operation

This operation can add rows, update rows, and remove rows all in a single transaction. It is a very generic tool that can be used to create behaviors like “insert if not exists”, “update or insert (i.e. upsert)”, or even replace a portion of existing data with new data (e.g. replace all data where month=“january”)

The merge insert operation works by combining new data from a source table with existing data in a target table by using a join. There are three categories of records.

“Matched” records are records that exist in both the source table and the target table. “Not matched” records exist only in the source table (e.g. these are new data) “Not matched by source” records exist only in the target table (this is old data)

The builder returned by this method can be used to customize what should happen for each category of data.

Please note that the data may appear to be reordered as part of this operation. This is because updated rows will be deleted from the dataset and then reinserted at the end with the new values.

Arguments

• on One or more columns to join on. This is how records from the source table and target table are matched. Typically this is some kind of key or id column.

Examples

let tmpdir = tempfile::tempdir().unwrap();
let db = lancedb::connect(tmpdir.path().to_str().unwrap())
    .execute()
    .await
    .unwrap();
let new_data = RecordBatchIterator::new(
    vec![RecordBatch::try_new(
        schema.clone(),
        vec![
            Arc::new(Int32Array::from_iter_values(0..10)),
            Arc::new(
                FixedSizeListArray::from_iter_primitive::<Float32Type, _, _>(
                    (0..10).map(|_| Some(vec![Some(1.0); 128])),
                    128,
                ),
            ),
        ],
    )
    .unwrap()]
    .into_iter()
    .map(Ok),
    schema.clone(),
);
// Perform an upsert operation
let mut merge_insert = tbl.merge_insert(&["id"]);
merge_insert
    .when_matched_update_all(None)
    .when_not_matched_insert_all();
merge_insert.execute(Box::new(new_data)).await.unwrap();

pub fn query(&self) -> Query

Create a Query Builder.

Queries allow you to search your existing data. By default the query will return all the data in the table in no particular order. The builder returned by this method can be used to control the query using filtering, vector similarity, sorting, and more.

Note: By default, all columns are returned. For best performance, you should only fetch the columns you need. See [Query::select_with_projection] for more details.

When appropriate, various indices and statistics will be used to accelerate the query.

Examples

Vector search

This example will find the 10 rows whose value in the “vector” column are closest to the query vector [1.0, 2.0, 3.0]. If an index has been created on the “vector” column then this will perform an ANN search.

The [Query::refine_factor] and [Query::nprobes] methods are used to control the recall / latency tradeoff of the search.

use crate::lancedb::Table;
use crate::lancedb::query::ExecutableQuery;
let stream = tbl
    .query()
    .nearest_to(&[1.0, 2.0, 3.0])
    .unwrap()
    .refine_factor(5)
    .nprobes(10)
    .execute()
    .await
    .unwrap();
let batches: Vec<RecordBatch> = stream.try_collect().await.unwrap();

SQL-style filter

This query will return up to 1000 rows whose value in the id column is greater than 5. LanceDb supports a broad set of filtering functions.

use crate::lancedb::Table;
use crate::lancedb::query::{ExecutableQuery, QueryBase};
let stream = tbl
    .query()
    .only_if("id > 5")
    .limit(1000)
    .execute()
    .await
    .unwrap();
let batches: Vec<RecordBatch> = stream.try_collect().await.unwrap();

Full scan

This query will return everything in the table in no particular order.

use crate::lancedb::Table;
use crate::lancedb::query::ExecutableQuery;
let stream = tbl.query().execute().await.unwrap();
let batches: Vec<RecordBatch> = stream.try_collect().await.unwrap();

pub fn vector_search(&self, query: impl IntoQueryVector) -> Result<VectorQuery>

Search the table with a given query vector.

This is a convenience method for preparing a vector query and is the same thing as calling nearest_to on the builder returned by query. See Query::nearest_to for more details.

pub async fn optimize(&self, action: OptimizeAction) -> Result<OptimizeStats>

Optimize the on-disk data and indices for better performance.

Modeled after VACUUM in PostgreSQL.

Optimization is discussed in more detail in the OptimizeAction documentation and covers three operations:

• Compaction: Merges small files into larger ones
• Prune: Removes old versions of the dataset
• Index: Optimizes the indices, adding new data to existing indices

Experimental API

The optimization process is undergoing active development and may change. Our goal with these changes is to improve the performance of optimization and reduce the complexity.

That being said, it is essential today to run optimize if you want the best performance. It should be stable and safe to use in production, but it our hope that the API may be simplified (or not even need to be called) in the future.

The frequency an application shoudl call optimize is based on the frequency of data modifications. If data is frequently added, deleted, or updated then optimize should be run frequently. A good rule of thumb is to run optimize if you have added or modified 100,000 or more records or run more than 20 data modification operations.

pub async fn add_columns( &self, transforms: NewColumnTransform, read_columns: Option<Vec<String>>, ) -> Result<AddColumnsResult>

Add new columns to the table, providing values to fill in.

pub async fn alter_columns( &self, alterations: &[ColumnAlteration], ) -> Result<AlterColumnsResult>

Change a column’s name or nullability.

pub async fn drop_columns(&self, columns: &[&str]) -> Result<DropColumnsResult>

Remove columns from the table.

pub async fn version(&self) -> Result<u64>

Retrieve the version of the table

LanceDb supports versioning. Every operation that modifies the table increases version. As long as a version hasn’t been deleted you can [Self::checkout] that version to view the data at that point. In addition, you can [Self::restore] the version to replace the current table with a previous version.

pub async fn checkout(&self, version: u64) -> Result<()>

Checks out a specific version of the Table

Any read operation on the table will now access the data at the checked out version. As a consequence, calling this method will disable any read consistency interval that was previously set.

This is a read-only operation that turns the table into a sort of “view” or “detached head”. Other table instances will not be affected. To make the change permanent you can use the [Self::restore] method.

Any operation that modifies the table will fail while the table is in a checked out state.

To return the table to a normal state use [Self::checkout_latest]

pub async fn checkout_tag(&self, tag: &str) -> Result<()>

Checks out a specific version of the Table by tag

Any read operation on the table will now access the data at the version referenced by the tag. As a consequence, calling this method will disable any read consistency interval that was previously set.

This is a read-only operation that turns the table into a sort of “view” or “detached head”. Other table instances will not be affected. To make the change permanent you can use the [Self::restore] method.

Any operation that modifies the table will fail while the table is in a checked out state.

To return the table to a normal state use [Self::checkout_latest]

pub async fn checkout_latest(&self) -> Result<()>

Ensures the table is pointing at the latest version

This can be used to manually update a table when the read_consistency_interval is None It can also be used to undo a [Self::checkout] operation

pub async fn restore(&self) -> Result<()>

Restore the table to the currently checked out version

This operation will fail if checkout has not been called previously

This operation will overwrite the latest version of the table with a previous version. Any changes made since the checked out version will no longer be visible.

Once the operation concludes the table will no longer be in a checked out state and the read_consistency_interval, if any, will apply.

pub async fn list_versions(&self) -> Result<Vec<Version>>

List all the versions of the table

pub async fn list_indices(&self) -> Result<Vec<IndexConfig>>

List all indices that have been created with Self::create_index

pub fn dataset_uri(&self) -> &str

Get the underlying dataset URI

Warning: This is an internal API and the return value is subject to change.

pub async fn index_stats( &self, index_name: impl AsRef<str>, ) -> Result<Option<IndexStatistics>>

Get statistics about an index. Returns None if the index does not exist.

pub async fn drop_index(&self, name: &str) -> Result<()>

Drop an index from the table.

Note: This is not yet available in LanceDB cloud.

This does not delete the index from disk, it just removes it from the table. To delete the index, run Self::optimize() after dropping the index.

Use Self::list_indices() to find the names of the indices.

pub async fn prewarm_index(&self, name: &str) -> Result<()>

Prewarm an index in the table

This is a hint to fully load the index into memory. It can be used to avoid cold starts

It is generally wasteful to call this if the index does not fit into the available cache.

Note: This function is not yet supported on all indices, in which case it may do nothing.

Use Self::list_indices() to find the names of the indices.

pub async fn wait_for_index( &self, index_names: &[&str], timeout: Duration, ) -> Result<()>

Poll until the columns are fully indexed. Will return Error::Timeout if the columns are not fully indexed within the timeout.

pub async fn tags(&self) -> Result<Box<dyn Tags + '_>>

Get the tags manager.

pub async fn stats(&self) -> Result<TableStatistics>

Retrieve statistics on the table

Trait Implementations

impl Clone for Table

fn clone(&self) -> Table

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Display for Table

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<NativeTable> for Table

fn from(table: NativeTable) -> Self

Converts to this type from the input type.