Enum OptimizeAction 

pub enum OptimizeAction {
    All,
    Compact {
        options: CompactionOptions,
        remap_options: Option<Arc<dyn IndexRemapperOptions>>,
    },
    Prune {
        older_than: Option<Duration>,
        delete_unverified: Option<bool>,
        error_if_tagged_old_versions: Option<bool>,
    },
    Index(OptimizeOptions),
}

Optimize the dataset.

Similar to VACUUM in PostgreSQL, it offers different options to optimize different parts of the table on disk.

By default, it optimizes everything, as OptimizeAction::All.

Variants

All

Run all optimizations with default values

Compact

Compacts files in the dataset

LanceDb uses a readonly filesystem for performance and safe concurrency. Every time new data is added it will be added into new files. Small files can hurt both read and write performance. Compaction will merge small files into larger ones.

All operations that modify data (add, delete, update, merge insert, etc.) will create new files. If these operations are run frequently then compaction should run frequently.

If these operations are never run (search only) then compaction is not necessary.

Fields

options: CompactionOptions

remap_options: Option<Arc<dyn IndexRemapperOptions>>

Prune

Prune old version of datasets

Every change in LanceDb is additive. When data is removed from a dataset a new version is created that doesn’t contain the removed data. However, the old version, which does contain the removed data, is left in place. This is necessary for consistency and concurrency and also enables time travel functionality like the ability to checkout an older version of the dataset to undo changes.

Over time, these old versions can consume a lot of disk space. The prune operation will remove versions of the dataset that are older than a certain age. This will free up the space used by that old data.

Once a version is pruned it can no longer be checked out.

Fields

older_than: Option<Duration>

The duration of time to keep versions of the dataset.

delete_unverified: Option<bool>

Because they may be part of an in-progress transaction, files newer than 7 days old are not deleted by default. If you are sure that there are no in-progress transactions, then you can set this to True to delete all files older than older_than.

WARNING: This should only be set to true if you can guarantee that no other process is currently working on this dataset. Otherwise the dataset could be put into a corrupted state.

error_if_tagged_old_versions: Option<bool>

If true, an error will be returned if there are any old versions that are still tagged.

Index(OptimizeOptions)

Optimize the indices

This operation optimizes all indices in the table. When new data is added to LanceDb it is not added to the indices. However, it can still turn up in searches because the search function will scan both the indexed data and the unindexed data in parallel. Over time, the unindexed data can become large enough that the search performance is slow. This operation will add the unindexed data to the indices without rerunning the full index creation process.

Optimizing an index is faster than re-training the index but it does not typically adjust the underlying model relied upon by the index. This can eventually lead to poor search accuracy and so users may still want to occasionally retrain the index after adding a large amount of data.

For example, when using IVF, an index will create clusters. Optimizing an index assigns unindexed data to the existing clusters, but it does not move the clusters or create new clusters.

Trait Implementations

impl Default for OptimizeAction

fn default() -> Self

Returns the “default value” for a type.