Struct DatasetWriteGuard 

pub struct DatasetWriteGuard<'a> { /* private fields */ }

Methods from Deref<Target = Dataset>

pub async fn checkout_version( &self, version: impl Into<Ref>, ) -> Result<Dataset, Error>

Check out a dataset version with a ref

pub fn tags(&self) -> Tags<'_>

pub fn branches(&self) -> Branches<'_>

pub async fn checkout_latest(&mut self) -> Result<(), Error>

Check out the latest version of the dataset

pub async fn checkout_branch(&self, branch: &str) -> Result<Dataset, Error>

Check out the latest version of the branch

pub async fn create_branch( &mut self, branch: &str, version: impl Into<Ref>, store_params: Option<ObjectStoreParams>, ) -> Result<Dataset, Error>

This is a two-phase operation:

• Create the branch dataset by shallow cloning.
• Create the branch metadata (a.k.a. BranchContents).

These two phases are not atomic. We consider BranchContents as the source of truth for the branch.

The cleanup procedure should:

• Clean up zombie branch datasets that have no related BranchContents.
• Delete broken BranchContents entries that have no related branch dataset.

If create_branch stops at phase 1, it may leave a zombie branch dataset, which can be cleaned up later. Such a zombie dataset may cause a branch creation failure if we use the same name to create_branch. In that case, you need to call force_delete_branch to interactively clean up the zombie dataset.

pub async fn delete_branch(&mut self, branch: &str) -> Result<(), Error>

pub async fn force_delete_branch(&mut self, branch: &str) -> Result<(), Error>

Delete the branch even if the BranchContents is not found. This could be useful when we have zombie branches and want to clean them up immediately.

pub async fn list_branches( &self, ) -> Result<HashMap<String, BranchContents>, Error>

pub async fn append( &mut self, batches: impl RecordBatchReader + Send + 'static, params: Option<WriteParams>, ) -> Result<(), Error>

Append to existing Dataset with a stream of RecordBatchs

Returns void result or Returns Error

pub fn uri(&self) -> &str

Get the fully qualified URI of this dataset.

pub fn branch_location(&self) -> BranchLocation

pub fn find_branch_location( &self, branch_name: &str, ) -> Result<BranchLocation, Error>

pub fn manifest(&self) -> &Manifest

Get the full manifest of the dataset version.

pub fn manifest_location(&self) -> &ManifestLocation

pub fn delta(&self) -> DatasetDeltaBuilder

Create a delta::DatasetDeltaBuilder to explore changes between dataset versions.

Example

let delta = dataset.delta()
    .compared_against_version(5)
    .build()?;
let inserted = delta.get_inserted_rows().await?;

pub async fn latest_manifest( &self, ) -> Result<(Arc<Manifest>, ManifestLocation), Error>

pub async fn read_transaction(&self) -> Result<Option<Transaction>, Error>

Read the transaction file for this version of the dataset.

If there was no transaction file written for this version of the dataset then this will return None.

pub async fn read_transaction_by_version( &self, version: u64, ) -> Result<Option<Transaction>, Error>

Read the transaction file for this version of the dataset.

If there was no transaction file written for this version of the dataset then this will return None.

pub async fn get_transactions( &self, recent_transactions: usize, ) -> Result<Vec<Option<Transaction>>, Error>

List transactions for the dataset, up to a maximum number.

This method iterates through dataset versions, starting from the current version, and collects the transaction for each version. It stops when either recent_transactions is reached or there are no more versions.

Arguments

• recent_transactions - Maximum number of transactions to return

Returns

A vector of optional transactions. Each element corresponds to a version, and may be None if no transaction file exists for that version.

pub async fn restore(&mut self) -> Result<(), Error>

Restore the currently checked out version of the dataset as the latest version.

pub fn cleanup_old_versions( &self, older_than: TimeDelta, delete_unverified: Option<bool>, error_if_tagged_old_versions: Option<bool>, ) -> Pin<Box<dyn Future<Output = Result<RemovalStats, Error>> + Send + '_>>

Removes old versions of the dataset from disk

This function will remove all versions of the dataset that are older than the provided timestamp. This function will not remove the current version of the dataset.

Once a version is removed it can no longer be checked out or restored. Any data unique to that version will be lost.

Arguments

• older_than - Versions older than this will be deleted.
• delete_unverified - If false (the default) then files will only be deleted if they are listed in at least one manifest. Otherwise these files will be kept since they cannot be distinguished from an in-progress transaction. Set to true to delete these files if you are sure there are no other in-progress dataset operations.

Returns

• RemovalStats - Statistics about the removal operation

pub fn cleanup_with_policy( &self, policy: CleanupPolicy, ) -> Pin<Box<dyn Future<Output = Result<RemovalStats, Error>> + Send + '_>>

Removes old versions of the dataset from storage

This function will remove all versions of the dataset that satisfies the given policy. This function will not remove the current version of the dataset.

Once a version is removed it can no longer be checked out or restored. Any data unique to that version will be lost.

Arguments

• policy - CleanupPolicy determines the behaviour of cleanup.

Returns

• RemovalStats - Statistics about the removal operation

pub fn scan(&self) -> Scanner

Create a Scanner to scan the dataset.

pub async fn count_rows(&self, filter: Option<String>) -> Result<usize, Error>

Count the number of rows in the dataset.

It offers a fast path of counting rows by just computing via metadata.

pub async fn take( &self, row_indices: &[u64], projection: impl Into<ProjectionRequest>, ) -> Result<RecordBatch, Error>

Take rows by indices.

pub async fn take_rows( &self, row_ids: &[u64], projection: impl Into<ProjectionRequest>, ) -> Result<RecordBatch, Error>

Take Rows by the internal ROW ids.

In Lance format, each row has a unique u64 id, which is used to identify the row globally.

let schema = dataset.schema().clone();
let row_ids = vec![0, 4, 7];
let rows = dataset.take_rows(&row_ids, schema).await.unwrap();

// We can have more fine-grained control over the projection, i.e., SQL projection.
let projection = ProjectionRequest::from_sql([("identity", "id * 2")]);
let rows = dataset.take_rows(&row_ids, projection).await.unwrap();

pub fn take_builder( self: &Arc<Dataset>, row_ids: &[u64], projection: impl Into<ProjectionRequest>, ) -> Result<TakeBuilder, Error>

pub async fn take_blobs( self: &Arc<Dataset>, row_ids: &[u64], column: impl AsRef<str>, ) -> Result<Vec<BlobFile>, Error>

Take BlobFile by row ids (row address).

pub async fn take_blobs_by_addresses( self: &Arc<Dataset>, row_addrs: &[u64], column: impl AsRef<str>, ) -> Result<Vec<BlobFile>, Error>

Take BlobFile by row addresses.

Row addresses are u64 values encoding (fragment_id << 32) | row_offset. Use this method when you already have row addresses, for example from a scan with with_row_address(). For row IDs (stable identifiers), use Self::take_blobs. For row indices (offsets), use Self::take_blobs_by_indices.

pub async fn take_blobs_by_indices( self: &Arc<Dataset>, row_indices: &[u64], column: impl AsRef<str>, ) -> Result<Vec<BlobFile>, Error>

Take BlobFile by row indices (offsets in the dataset).

pub fn take_scan( &self, row_ranges: Pin<Box<dyn Stream<Item = Result<Range<u64>, Error>> + Send>>, projection: Arc<Schema>, batch_readahead: usize, ) -> DatasetRecordBatchStream

Get a stream of batches based on iterator of ranges of row numbers.

This is an experimental API. It may change at any time.

pub async fn delete(&mut self, predicate: &str) -> Result<(), Error>

Delete rows based on a predicate.

pub async fn add_bases( self: &Arc<Dataset>, new_bases: Vec<BasePath>, transaction_properties: Option<HashMap<String, String>>, ) -> Result<Dataset, Error>

Add new base paths to the dataset.

This method allows you to register additional storage locations (buckets) that can be used for future data writes. The base paths are added to the dataset’s manifest and can be referenced by name in subsequent write operations.

Arguments

• new_bases - A vector of lance_table::format::BasePath objects representing the new storage locations to add. Each base path should have a unique name and path.

Returns

Returns a new Dataset instance with the updated manifest containing the new base paths.

pub async fn count_deleted_rows(&self) -> Result<usize, Error>

pub fn object_store(&self) -> &ObjectStore

pub fn storage_options(&self) -> Option<&HashMap<String, String>>

Returns the storage options used when opening this dataset, if any.

pub fn storage_options_provider( &self, ) -> Option<Arc<dyn StorageOptionsProvider>>

Returns the storage options provider used when opening this dataset, if any.

pub fn data_dir(&self) -> Path

pub fn indices_dir(&self) -> Path

pub fn session(&self) -> Arc<Session>

pub fn version(&self) -> Version

pub async fn index_cache_entry_count(&self) -> usize

Get the number of entries currently in the index cache.

pub async fn index_cache_hit_rate(&self) -> f32

Get cache hit ratio.

pub fn cache_size_bytes(&self) -> u64

pub async fn versions(&self) -> Result<Vec<Version>, Error>

Get all versions.

pub async fn latest_version_id(&self) -> Result<u64, Error>

Get the latest version of the dataset This is meant to be a fast path for checking if a dataset has changed. This is why we don’t return the full version struct.

pub fn count_fragments(&self) -> usize

pub fn schema(&self) -> &Schema

Get the schema of the dataset

pub fn empty_projection(self: &Arc<Dataset>) -> Projection

Similar to Self::schema, but only returns fields that are not marked as blob columns Creates a new empty projection into the dataset schema

pub fn full_projection(self: &Arc<Dataset>) -> Projection

Creates a projection that includes all columns in the dataset

pub fn get_fragments(&self) -> Vec<FileFragment>

Get fragments.

If filter is provided, only fragments with the given name will be returned.

pub fn get_fragment(&self, fragment_id: usize) -> Option<FileFragment>

pub fn fragments(&self) -> &Arc<Vec<Fragment>>

pub fn get_frags_from_ordered_ids( &self, ordered_ids: &[u32], ) -> Vec<Option<FileFragment>>

pub async fn num_small_files(&self, max_rows_per_group: usize) -> usize

Gets the number of files that are so small they don’t even have a full group. These are considered too small because reading many of them is much less efficient than reading a single file because the separate files split up what would otherwise be single IO requests into multiple.

pub async fn validate(&self) -> Result<(), Error>

pub async fn migrate_manifest_paths_v2(&mut self) -> Result<(), Error>

Migrate the dataset to use the new manifest path scheme.

This function will rename all V1 manifests to ManifestNamingScheme::V2. These paths provide more efficient opening of datasets with many versions on object stores.

This function is idempotent, and can be run multiple times without changing the state of the object store.

However, it should not be run while other concurrent operations are happening. And it should also run until completion before resuming other operations.

let mut dataset = Dataset::write(data, "memory://test", None).await.unwrap();
assert_eq!(dataset.manifest_location().naming_scheme, ManifestNamingScheme::V1);

dataset.migrate_manifest_paths_v2().await.unwrap();
assert_eq!(dataset.manifest_location().naming_scheme, ManifestNamingScheme::V2);

pub async fn shallow_clone( &mut self, target_path: &str, version: impl Into<Ref>, store_params: Option<ObjectStoreParams>, ) -> Result<Dataset, Error>

Shallow clone the target version into a new dataset at target_path. ‘target_path’: the uri string to clone the dataset into. ‘version’: the version cloned from, could be a version number or tag. ‘store_params’: the object store params to use for the new dataset.

pub fn sql(&self, sql: &str) -> SqlQueryBuilder

Run a SQL query against the dataset. The underlying SQL engine is DataFusion. Please refer to the DataFusion documentation for supported SQL syntax.

pub async fn add_columns( &mut self, transforms: NewColumnTransform, read_columns: Option<Vec<String>>, batch_size: Option<u32>, ) -> Result<(), Error>

Append new columns to the dataset.

pub async fn alter_columns( &mut self, alterations: &[ColumnAlteration], ) -> Result<(), Error>

Modify columns in the dataset, changing their name, type, or nullability.

If only changing the name or nullability of a column, this is a zero-copy operation and any indices will be preserved. If changing the type of a column, the data for that column will be rewritten and any indices will be dropped. The old column data will not be immediately deleted. To remove it, call optimize::compact_files() and then cleanup::cleanup_old_versions() on the dataset.

pub async fn drop_columns(&mut self, columns: &[&str]) -> Result<(), Error>

Remove columns from the dataset.

This is a metadata-only operation and does not remove the data from the underlying storage. In order to remove the data, you must subsequently call optimize::compact_files() to rewrite the data without the removed columns and then call cleanup::cleanup_old_versions() to remove the old files.

pub async fn drop(&mut self, columns: &[&str]) -> Result<(), Error>

👎Deprecated since 0.9.12: Please use drop_columns instead.

Drop columns from the dataset and return updated dataset. Note that this is a zero-copy operation and column is not physically removed from the dataset. Parameters:

• columns: the list of column names to drop.

pub async fn merge( &mut self, stream: impl RecordBatchReader + Send + 'static, left_on: &str, right_on: &str, ) -> Result<(), Error>

Merge this dataset with another arrow Table / Dataset, and returns a new version of dataset.

Parameters:

• stream: the stream of RecordBatch to merge.
• left_on: the column name to join on the left side (self).
• right_on: the column name to join on the right side (stream).

Returns: a new version of dataset.

It performs a left-join on the two datasets.

pub fn metadata(&self) -> &HashMap<String, String>

Get dataset metadata.

pub fn config(&self) -> &HashMap<String, String>

Get the dataset config from manifest

pub async fn delete_config_keys( &mut self, delete_keys: &[&str], ) -> Result<(), Error>

👎Deprecated: Use the new update_config(values, replace) method - pass None values to delete keys

Delete keys from the config.

pub fn update_metadata( &mut self, values: impl IntoIterator<Item = impl Into<UpdateMapEntry>>, ) -> UpdateMetadataBuilder<'_>

Update table metadata.

Pass None for a value to remove that key.

Use .replace() to replace the entire metadata map instead of merging.

Returns the updated metadata map after the operation.

// Update single key
dataset.update_metadata([("key", "value")]).await?;

// Remove a key
dataset.update_metadata([("to_delete", None)]).await?;

// Clear all metadata
dataset.update_metadata([] as [UpdateMapEntry; 0]).replace().await?;

// Replace full metadata
dataset.update_metadata([("k1", "v1"), ("k2", "v2")]).replace().await?;

pub fn update_config( &mut self, values: impl IntoIterator<Item = impl Into<UpdateMapEntry>>, ) -> UpdateMetadataBuilder<'_>

Update config.

Pass None for a value to remove that key.

Use .replace() to replace the entire config map instead of merging.

Returns the updated config map after the operation.

// Update single key
dataset.update_config([("key", "value")]).await?;

// Remove a key
dataset.update_config([("to_delete", None)]).await?;

// Clear all config
dataset.update_config([] as [UpdateMapEntry; 0]).replace().await?;

// Replace full config
dataset.update_config([("k1", "v1"), ("k2", "v2")]).replace().await?;

pub fn update_schema_metadata( &mut self, values: impl IntoIterator<Item = impl Into<UpdateMapEntry>>, ) -> UpdateMetadataBuilder<'_>

Update schema metadata.

Pass None for a value to remove that key.

Use .replace() to replace the entire schema metadata map instead of merging.

Returns the updated schema metadata map after the operation.

// Update single key
dataset.update_schema_metadata([("key", "value")]).await?;

// Remove a key
dataset.update_schema_metadata([("to_delete", None)]).await?;

// Clear all schema metadata
dataset.update_schema_metadata([] as [UpdateMapEntry; 0]).replace().await?;

// Replace full schema metadata
dataset.update_schema_metadata([("k1", "v1"), ("k2", "v2")]).replace().await?;

pub async fn replace_schema_metadata( &mut self, new_values: impl IntoIterator<Item = (String, String)>, ) -> Result<(), Error>

👎Deprecated: Use the new update_schema_metadata(values).replace() instead

Update schema metadata

pub fn update_field_metadata(&mut self) -> UpdateFieldMetadataBuilder<'_>

Update field metadata

// Update metadata by field path
dataset.update_field_metadata()
    .update("path.to_field", [("key", "value")])?
    .await?;

// Update metadata by field id
dataset.update_field_metadata()
    .update(12, [("key", "value")])?
    .await?;

// Clear field metadata
dataset.update_field_metadata()
    .replace("path.to_field", [] as [UpdateMapEntry; 0])?
    .replace(12, [] as [UpdateMapEntry; 0])?
    .await?;

// Replace field metadata
dataset.update_field_metadata()
    .replace("field_name", [("k1", "v1"), ("k2", "v2")])?
    .await?;

pub async fn replace_field_metadata( &mut self, new_values: impl IntoIterator<Item = (u32, HashMap<String, String>)>, ) -> Result<(), Error>

Update field metadata

Trait Implementations

impl Deref for DatasetWriteGuard<'_>

type Target = Dataset

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl DerefMut for DatasetWriteGuard<'_>

fn deref_mut(&mut self) -> &mut Self::Target

Mutably dereferences the value.