Struct DatasetBuilder 

pub struct DatasetBuilder { /* private fields */ }

builder for loading a Dataset.

Implementations

impl DatasetBuilder

pub fn from_uri<T: AsRef<str>>(table_uri: T) -> Self

pub async fn from_namespace( namespace: Arc<dyn LanceNamespace>, table_id: Vec<String>, ignore_namespace_table_storage_options: bool, ) -> Result<Self>

Create a DatasetBuilder from a LanceNamespace

This will automatically fetch the table location and storage options from the namespace via describe_table().

Storage options from the namespace will override any user-provided storage options set via .with_storage_options(). This ensures the namespace is always the source of truth for storage options.

Arguments

• namespace - The namespace implementation to fetch table info from
• table_id - The table identifier (e.g., vec![“my_table”])
• ignore_namespace_table_storage_options - If true, storage options returned from the namespace’s describe_table() will be ignored (treated as None). Defaults to false.

Example

use lance_namespace_impls::ConnectBuilder;
use lance::dataset::DatasetBuilder;

// Connect to a REST namespace
let namespace = ConnectBuilder::new("rest")
    .property("uri", "http://localhost:8080")
    .connect()
    .await?;

// Load a dataset using storage options from namespace
let dataset = DatasetBuilder::from_namespace(
    namespace.clone(),
    vec!["my_table".to_string()],
    false,
)
.await?
.load()
.await?;

// Load a dataset ignoring namespace storage options
let dataset = DatasetBuilder::from_namespace(
    namespace,
    vec!["my_table".to_string()],
    true,
)
.await?
.load()
.await?;

impl DatasetBuilder

pub fn with_index_cache_size_bytes(self, cache_size: usize) -> Self

Set the cache size for indices. Set to zero, to disable the cache.

pub fn with_index_cache_size(self, cache_size: usize) -> Self

👎Deprecated since 0.30.0: Use with_index_cache_size_bytes instead

Set the cache size for indices. Set to zero, to disable the cache.

pub fn with_metadata_cache_size_bytes(self, cache_size: usize) -> Self

Size of the metadata cache in bytes. This cache stores metadata in memory for faster open table and scans. The default is 1 GiB.

pub fn with_metadata_cache_size(self, cache_size: usize) -> Self

👎Deprecated since 0.30.0: Use with_metadata_cache_size_bytes instead

Set the cache size for the file metadata. Set to zero to disable this cache.

pub fn with_block_size(self, block_size: usize) -> Self

The block size passed to the underlying Object Store reader.

This is used to control the minimal request size. Defaults to 4KB for local files and 64KB for others

pub fn with_version(self, version: u64) -> Self

Sets version for the builder using a version number

pub fn with_branch(self, branch: &str, version_number: Option<u64>) -> Self

Sets version for the builder using a branch and optional version number If version_number is null, checkout the latest version

pub fn with_tag(self, tag: &str) -> Self

Sets version for the builder using a tag

pub fn with_commit_handler(self, commit_handler: Arc<dyn CommitHandler>) -> Self

pub fn with_s3_credentials_refresh_offset(self, offset: Duration) -> Self

Sets the s3 credentials refresh. This only applies to s3 storage.

pub fn with_object_store( self, object_store: Arc<DynObjectStore>, location: Url, commit_handler: Arc<dyn CommitHandler>, ) -> Self

👎Deprecated: Implement an ObjectStoreProvider instead

Directly set the object store to use.

pub fn with_serialized_manifest(self, manifest: &[u8]) -> Result<Self>

Use a serialized manifest instead of loading it from the object store.

This is common when transferring a dataset across IPC boundaries.

pub fn with_storage_options( self, storage_options: HashMap<String, String>, ) -> Self

Set options used to initialize storage backend

Options may be passed in the HashMap or set as environment variables. See documentation of underlying object store implementation for details.

• Azure options
• S3 options
• Google options

pub fn with_storage_option( self, key: impl AsRef<str>, value: impl AsRef<str>, ) -> Self

Set a single option used to initialize storage backend For example, to set the region for S3, you can use:

let builder = DatasetBuilder::from_uri("s3://bucket/path")
    .with_storage_option("region", "us-east-1");

pub fn with_storage_options_provider( self, provider: Arc<dyn StorageOptionsProvider>, ) -> Self

Enable credential vending from a LanceNamespace

Credentials will be automatically refreshed from the namespace before they expire. The namespace should return expires_at_millis in the storage_options from describe_table().

Use with_s3_credentials_refresh_offset() to configure how early credentials should be refreshed before they expire (default is 5 minutes).

Arguments

• provider - The storage options provider to fetch credentials from

Example

use std::sync::Arc;
use std::time::Duration;
use lance_namespace_impls::ConnectBuilder;
use lance_io::object_store::{StorageOptionsProvider, LanceNamespaceStorageOptionsProvider};

// Connect to a REST namespace
let namespace = ConnectBuilder::new("rest")
    .property("uri", "http://localhost:8080")
    .connect()
    .await?;

// Create a storage options provider from namespace
let provider = Arc::new(LanceNamespaceStorageOptionsProvider::new(
    namespace,
    vec!["my_table".to_string()],
));

// With default settings (5 minute refresh offset)
let dataset = DatasetBuilder::from_uri("s3://bucket/table.lance")
    .with_storage_options_provider(provider)
    .load()
    .await?;

// With custom refresh offset (refresh 10 minutes before expiration) let dataset = DatasetBuilder::from_uri(“s3://bucket/table.lance”) .with_storage_options_provider(provider.clone()) .with_s3_credentials_refresh_offset(Duration::from_secs(600)) .load() .await?;

pub fn with_read_params(self, read_params: ReadParams) -> Self

Set options based on ReadParams.

pub fn with_write_params(self, write_params: WriteParams) -> Self

Set options based on WriteParams.

pub fn with_session(self, session: Arc<Session>) -> Self

Re-use an existing session.

The session holds caches for index and metadata.

If this is set, then with_index_cache_size and with_metadata_cache_size are ignored.

pub async fn build_object_store( self, ) -> Result<(Arc<ObjectStore>, Path, Arc<dyn CommitHandler>)>

Build a lance object store for the given config

pub async fn load(self) -> Result<Dataset>

pub async fn load_by_uri( session: Arc<Session>, manifest: Option<Manifest>, file_reader_options: Option<FileReaderOptions>, table_uri: String, version_number: Option<u64>, object_store: Arc<ObjectStore>, base_path: Path, commit_handler: Arc<dyn CommitHandler>, store_params: Option<ObjectStoreParams>, ) -> Result<Dataset>

Trait Implementations

impl Clone for DatasetBuilder

fn clone(&self) -> DatasetBuilder

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DatasetBuilder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.