Struct DatasetReadGuard 

pub struct DatasetReadGuard<'a> { /* private fields */ }

Methods from Deref<Target = Dataset>

pub async fn tracked_files( &self, ) -> Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch, DataFusionError>> + Send>>

Returns one row per (version, file) for every file referenced in any manifest.

Each row contains the manifest version, the storage root URI, the file path relative to that URI, and the file type.

Schema

Column	Type	Notes
version	Int64 (non-null)	Manifest version number
base_uri	Dictionary(Int32, Utf8) (non-null)	Storage root for this file
path	Utf8 (non-null)	Relative to base_uri
type	Dictionary(Int8, Utf8) (non-null)	One of: data file, manifest, deletion file, transaction file, index file

Output order is non-deterministic.

pub async fn tracked_files_with_options( &self, options: TrackedFilesOptions, ) -> Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch, DataFusionError>> + Send>>

Like Self::tracked_files, but with additional options for filtering and progress reporting.

pub async fn all_files( &self, ) -> Pin<Box<dyn RecordBatchStream<Item = Result<RecordBatch, DataFusionError>> + Send>>

Returns one row per file that physically exists at the dataset’s base URI.

This scans the primary object store root only. Additional base_paths entries in the manifest (for externally-located data files) are not scanned by this method.

Schema

Column	Type	Notes
base_uri	Dictionary(Int32, Utf8) (non-null)	Storage root
path	Utf8 (non-null)	Relative to base_uri
size_bytes	Int64 (non-null)	File size in bytes
last_modified	Timestamp(Microsecond, "UTC") (non-null)	Last modification time

pub async fn checkout_version( &self, version: impl Into<Ref>, ) -> Result<Dataset, Error>

Check out a dataset version with a ref

pub fn tags(&self) -> Tags<'_>

pub fn statistics(&self) -> DatasetStatistics<'_>

A handle for cheap, index-derived statistics about this dataset (e.g. a column’s global value range) that never scan data.

pub fn branches(&self) -> Branches<'_>

pub async fn checkout_branch(&self, branch: &str) -> Result<Dataset, Error>

Check out the latest version of the branch

pub async fn list_branches( &self, ) -> Result<HashMap<String, BranchContents>, Error>

pub fn uri(&self) -> &str

Get the fully qualified URI of this dataset.

pub fn branch_location(&self) -> BranchLocation

pub async fn branch_identifier(&self) -> Result<BranchIdentifier, Error>

pub fn manifest(&self) -> &Manifest

Get the full manifest of the dataset version.

pub fn manifest_location(&self) -> &ManifestLocation

pub fn delta(&self) -> DatasetDeltaBuilder

Create a delta::DatasetDeltaBuilder to explore changes between dataset versions.

Example

let delta = dataset.delta()
    .compared_against_version(5)
    .build()?;
let inserted = delta.get_inserted_rows().await?;

pub async fn latest_manifest( &self, ) -> Result<(Arc<Manifest>, ManifestLocation), Error>

pub async fn read_transaction(&self) -> Result<Option<Transaction>, Error>

Read the transaction file for this version of the dataset.

If there was no transaction file written for this version of the dataset then this will return None.

pub async fn read_version_transaction( &self, version: u64, ) -> Result<VersionTransaction, Error>

Read the transaction (if any) and commit timestamp of a version of the dataset. version is a version number on this dataset’s current branch.

Reads the version’s manifest transiently: no historical Dataset is constructed, no IndexSection is decoded, and no session cache is read or written, so scanning many historical versions does not fill the shared caches.

Returns an error if the version does not exist (for example, if it has been cleaned up).

Example

let record = dataset.read_version_transaction(5).await?;
let committed_at = record.timestamp;
let operation = record.transaction.as_ref().map(|t| t.operation.name());

pub async fn read_transaction_by_version( &self, version: u64, ) -> Result<Option<Transaction>, Error>

Read the transaction file for this version of the dataset.

If there was no transaction file written for this version of the dataset then this will return None.

Does not populate the session caches; see Self::read_version_transaction.

Example

let transaction = dataset.read_transaction_by_version(5).await?;
let operation = transaction.as_ref().map(|t| t.operation.name());

pub async fn get_transactions( &self, recent_transactions: usize, ) -> Result<Vec<Option<Transaction>>, Error>

List transactions for the dataset, up to a maximum number.

This method iterates through dataset versions, starting from the current version, and collects the transaction for each version. It stops when either recent_transactions is reached or there are no more versions.

Arguments

• recent_transactions - Maximum number of transactions to return

Returns

A vector of optional transactions. Each element corresponds to a version, and may be None if no transaction file exists for that version.

pub fn cleanup_old_versions( &self, older_than: TimeDelta, delete_unverified: Option<bool>, error_if_tagged_old_versions: Option<bool>, ) -> Pin<Box<dyn Future<Output = Result<RemovalStats, Error>> + Send + '_>>

Removes old versions of the dataset from disk

This function will remove all versions of the dataset that are older than the provided timestamp. This function will not remove the current version of the dataset.

Once a version is removed it can no longer be checked out or restored. Any data unique to that version will be lost.

Arguments

• older_than - Versions older than this will be deleted.
• delete_unverified - If false (the default) then files will only be deleted if they are listed in at least one manifest. Otherwise these files will be kept since they cannot be distinguished from an in-progress transaction. Set to true to delete these files if you are sure there are no other in-progress dataset operations.

Returns

• RemovalStats - Statistics about the removal operation

pub fn cleanup_with_policy( &self, policy: CleanupPolicy, ) -> Pin<Box<dyn Future<Output = Result<RemovalStats, Error>> + Send + '_>>

Removes old versions of the dataset from storage

This function will remove all versions of the dataset that satisfies the given policy. This function will not remove the current version of the dataset.

Once a version is removed it can no longer be checked out or restored. Any data unique to that version will be lost.

Arguments

• policy - CleanupPolicy determines the behaviour of cleanup.

Returns

• RemovalStats - Statistics about the removal operation

pub fn cleanup(&self, policy: CleanupPolicy) -> CleanupOperation<'_>

Creates a cleanup operation for this dataset.

The returned operation can be explained without deleting files, or executed to re-evaluate the current dataset state and remove files.

pub fn scan(&self) -> Scanner

Create a Scanner to scan the dataset.

pub async fn count_rows(&self, filter: Option<String>) -> Result<usize, Error>

Count the number of rows in the dataset.

It offers a fast path of counting rows by just computing via metadata.

pub async fn take( &self, row_indices: &[u64], projection: impl Into<ProjectionRequest>, ) -> Result<RecordBatch, Error>

Take rows by indices.

pub async fn take_rows( &self, row_ids: &[u64], projection: impl Into<ProjectionRequest>, ) -> Result<RecordBatch, Error>

Take Rows by the internal ROW ids.

In Lance format, each row has a unique u64 id, which is used to identify the row globally.

let schema = dataset.schema().clone();
let row_ids = vec![0, 4, 7];
let rows = dataset.take_rows(&row_ids, schema).await.unwrap();

// We can have more fine-grained control over the projection, i.e., SQL projection.
let projection = ProjectionRequest::from_sql([("identity", "id * 2")]);
let rows = dataset.take_rows(&row_ids, projection).await.unwrap();

pub fn take_builder( self: &Arc<Dataset>, row_ids: &[u64], projection: impl Into<ProjectionRequest>, ) -> Result<TakeBuilder, Error>

pub async fn take_blobs( self: &Arc<Dataset>, row_ids: &[u64], column: impl AsRef<str>, ) -> Result<Vec<Option<BlobFile>>, Error>

Take BlobFile by row IDs.

The returned vector has one element per row ID. Null blob values are represented as None; valid empty blobs return a BlobFile with size zero.

let blobs = dataset.take_blobs(&[42], "images").await?;
match &blobs[0] {
    None => { /* The selected blob is null. */ }
    Some(blob) if blob.size() == 0 => { /* The selected blob is valid but empty. */ }
    Some(blob) => { let _size = blob.size(); }
}

pub async fn take_blobs_by_addresses( self: &Arc<Dataset>, row_addrs: &[u64], column: impl AsRef<str>, ) -> Result<Vec<Option<BlobFile>>, Error>

Take BlobFile by row addresses.

Row addresses are u64 values encoding (fragment_id << 32) | row_offset. Use this method when you already have row addresses, for example from a scan with with_row_address(). For row IDs (stable identifiers), use Self::take_blobs. For row indices (offsets), use Self::take_blobs_by_indices. The result has the same null and empty blob representation as Self::take_blobs.

let blobs = dataset
    .take_blobs_by_addresses(&[row_address], "images")
    .await?;
match &blobs[0] {
    None => { /* The selected blob is null. */ }
    Some(blob) if blob.size() == 0 => { /* The selected blob is valid but empty. */ }
    Some(blob) => { let _size = blob.size(); }
}

pub async fn take_blobs_by_indices( self: &Arc<Dataset>, row_indices: &[u64], column: impl AsRef<str>, ) -> Result<Vec<Option<BlobFile>>, Error>

Take BlobFile by row indices (offsets in the dataset).

The result has the same null and empty blob representation as Self::take_blobs.

let blobs = dataset.take_blobs_by_indices(&[0], "images").await?;
match &blobs[0] {
    None => { /* The selected blob is null. */ }
    Some(blob) if blob.size() == 0 => { /* The selected blob is valid but empty. */ }
    Some(blob) => { let _size = blob.size(); }
}

pub fn read_blobs( self: &Arc<Dataset>, column: impl AsRef<str>, ) -> Result<ReadBlobsBuilder, Error>

Create a planned blob reader for a blob column.

This API complements Self::take_blobs. take_blobs returns BlobFile handles for caller-driven random access, while read_blobs builds a streaming read plan for sequential or batched blob retrieval. Every selected row produces one result: null blob values have ReadBlob::data set to None, while valid empty blobs contain an empty buffer.

let blobs = dataset
    .read_blobs("images")?
    .with_row_indices(vec![0, 1, 2])
    .execute()
    .await?;

pub fn read_blob_ranges( self: &Arc<Dataset>, column: impl AsRef<str>, ) -> Result<ReadBlobRangesBuilder, Error>

Create a planned reader for row-specific blob-local byte ranges.

Each BlobRangeRequest contains both its row selector and byte range, so requests can be repeated or reordered without coordinating parallel selector and range lists. Every request produces one result. A null blob has ReadBlobRange::data set to None; an empty range on a non-null blob contains an empty buffer.

let ranges = dataset
    .read_blob_ranges("images")?
    .with_row_indices([
        BlobRangeRequest::new(7, 0, 1024),
        BlobRangeRequest::new(7, 4096, 1024),
    ])
    .execute()
    .await?;

pub fn take_scan( &self, row_ranges: Pin<Box<dyn Stream<Item = Result<Range<u64>, Error>> + Send>>, projection: Arc<Schema>, batch_readahead: usize, ) -> DatasetRecordBatchStream

Get a stream of batches based on iterator of ranges of row numbers.

This is an experimental API. It may change at any time.

pub async fn sample( &self, n: usize, projection: &Schema, fragment_ids: Option<&[u32]>, ) -> Result<RecordBatch, Error>

Randomly sample n rows from the dataset.

If fragment_ids is provided, sampling is limited to rows from those fragments in the current dataset version.

The returned rows are in row-id order (not random order), which allows the underlying take operation to use an efficient sorted code path.

pub async fn add_bases( self: &Arc<Dataset>, new_bases: Vec<BasePath>, transaction_properties: Option<HashMap<String, String>>, ) -> Result<Dataset, Error>

Add new base paths to the dataset.

This method allows you to register additional storage locations (buckets) that can be used for future data writes. The base paths are added to the dataset’s manifest and can be referenced by name in subsequent write operations.

Arguments

• new_bases - A vector of lance_table::format::BasePath objects representing the new storage locations to add. Each base path should have a unique name and path.

Returns

Returns a new Dataset instance with the updated manifest containing the new base paths.

pub async fn count_deleted_rows(&self) -> Result<usize, Error>

pub fn with_object_store( &self, object_store: Arc<ObjectStore>, store_params: Option<ObjectStoreParams>, ) -> Dataset

Clone this dataset with a different object store binding.

The returned dataset shares metadata, session state, and caches with the original dataset, but all subsequent operations on the returned dataset use the supplied object store.

pub fn with_object_store_wrappers( &self, wrappers: impl IntoIterator<Item = Arc<dyn WrappingObjectStore>>, ) -> Dataset

Clone this dataset with extra object store wrappers applied to all read stores.

The returned dataset uses the wrappers for the already-open primary object store and appends the same wrappers to the dataset-level and base-specific object store params used when additional base stores are opened later.

pub fn storage_options(&self) -> Option<&HashMap<String, String>>

👎Deprecated since 0.25.0:

Use initial_storage_options() instead

Returns the initial storage options used when opening this dataset, if any.

This returns the static initial options without triggering any refresh. For the latest refreshed options, use Self::latest_storage_options.

pub fn initial_storage_options(&self) -> Option<&HashMap<String, String>>

Returns the initial storage options without triggering any refresh.

For the latest refreshed options, use Self::latest_storage_options.

pub fn storage_options_provider( &self, ) -> Option<Arc<dyn StorageOptionsProvider>>

Returns the storage options provider used when opening this dataset, if any.

pub fn storage_options_accessor(&self) -> Option<Arc<StorageOptionsAccessor>>

Returns the unified storage options accessor for this dataset, if any.

The accessor handles both static and dynamic storage options with automatic caching and refresh. Use StorageOptionsAccessor::get_storage_options to get the latest options.

pub async fn latest_storage_options( &self, ) -> Result<Option<StorageOptions>, Error>

Returns the latest (possibly refreshed) storage options.

If a dynamic storage options provider is configured, this will return the cached options if still valid, or fetch fresh options if expired.

For the initial static options without refresh, use Self::storage_options.

Returns

• Ok(Some(options)) - Storage options are available (static or refreshed)
• Ok(None) - No storage options were configured for this dataset
• Err(...) - Error occurred while fetching/refreshing options from provider

pub fn data_dir(&self) -> Path

pub fn indices_dir(&self) -> Path

pub fn transactions_dir(&self) -> Path

pub fn deletions_dir(&self) -> Path

pub fn versions_dir(&self) -> Path

pub async fn create_data_file( &self, path: &str, base_id: Option<u32>, ) -> Result<DataFile, Error>

Create a DataFile by reading metadata from an existing lance file.

This reads the file’s schema and version information, matches columns to the dataset’s schema to determine field IDs, and calculates column indices. This is useful for constructing DataFile metadata needed for operations like Operation::DataReplacement.

Arguments

• path - The path to the data file, relative to the dataset’s data directory.
• base_id - The base path ID if the file is outside the dataset directory.

pub async fn object_store( &self, base_id: Option<u32>, ) -> Result<Arc<ObjectStore>, Error>

Resolve the object store for the primary dataset or an additional base.

Pass None to get the primary dataset object store. Pass Some(base_id) when resolving a file whose metadata references an additional base.

pub fn store_params(&self) -> Option<&ObjectStoreParams>

The ObjectStoreParams this dataset was opened with, or None when opened without explicit params. Lets a caller re-open a derived path (e.g. a MemWAL SSTable) with the same store this dataset used.

pub fn session(&self) -> Arc<Session>

pub fn version_id(&self) -> u64

Get the currently checked-out version id.

This is a cheap accessor that reads the id directly from the loaded manifest without constructing the full Version summary.

pub fn version(&self) -> Version

Get the currently checked-out version details.

This constructs a full Version, including summary metadata derived from the loaded manifest fragments.

pub async fn index_cache_entry_count(&self) -> usize

Get the number of entries currently in the index cache.

pub async fn index_cache_hit_rate(&self) -> f32

Get cache hit ratio.

pub fn cache_size_bytes(&self) -> u64

pub async fn versions(&self) -> Result<Vec<Version>, Error>

Get all versions.

pub async fn list_detached_manifests( &self, ) -> Result<Vec<ManifestLocation>, Error>

List all detached manifest locations.

Detached manifests are versions that are not part of the main version history. They are created by commit_detached and can be used for staging changes.

To read transaction properties from a detached manifest:

let detached = dataset.list_detached_manifests().await?;
for location in detached {
    let ds = dataset.checkout_version(location.version).await?;
    let tx = ds.read_transaction().await?;
    // Access tx.transaction_properties
}

pub async fn latest_version_id(&self) -> Result<u64, Error>

Get the latest version of the dataset This is meant to be a fast path for checking if a dataset has changed. This is why we don’t return the full version struct.

pub async fn is_stale(&self) -> Result<bool, Error>

Return whether the dataset has a newer committed version.

pub fn count_fragments(&self) -> usize

pub fn schema(&self) -> &Schema

Get the schema of the dataset

pub fn empty_projection(self: &Arc<Dataset>) -> Projection

Similar to Self::schema, but only returns fields that are not marked as blob columns Creates a new empty projection into the dataset schema

pub fn full_projection(self: &Arc<Dataset>) -> Projection

Creates a projection that includes all columns in the dataset

pub fn get_fragments(&self) -> Vec<FileFragment>

Get fragments.

pub fn iter_fragments(&self) -> impl Iterator<Item = &Fragment>

Iterate over manifest fragments without allocating FileFragment wrappers.

pub fn get_fragment(&self, fragment_id: usize) -> Option<FileFragment>

pub fn fragments(&self) -> &Arc<Vec<Fragment>>

pub fn get_frags_from_ordered_ids( &self, ordered_ids: &[u32], ) -> Vec<Option<FileFragment>>

Resolves fragments for the given ids without scanning the manifest.

The ids do not need to be sorted or deduplicated. Each id is resolved independently via the fragment bitmap.

pub async fn num_small_files(&self, max_rows_per_group: usize) -> usize

Gets the number of files that are so small they don’t even have a full group. These are considered too small because reading many of them is much less efficient than reading a single file because the separate files split up what would otherwise be single IO requests into multiple.

pub async fn validate(&self) -> Result<(), Error>

pub fn sql(&self, sql: &str) -> SqlQueryBuilder

Run a SQL query against the dataset. The underlying SQL engine is DataFusion. Please refer to the DataFusion documentation for supported SQL syntax.

pub async fn merge_index_metadata( &self, index_uuid: &Uuid, index_type: IndexType, _batch_readhead: Option<usize>, progress: Arc<dyn IndexBuildProgress>, ) -> Result<(), Error>

Merge a distributed scalar index into a single root artifact and report progress via the supplied callback.

pub fn metadata(&self) -> &HashMap<String, String>

Get dataset metadata.

pub fn config(&self) -> &HashMap<String, String>

Get the dataset config from manifest

Trait Implementations

impl Deref for DatasetReadGuard<'_>

type Target = Dataset

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.