Struct ObjectStore 

pub struct ObjectStore {
    pub inner: Arc<dyn OSObjectStore>,
    pub use_constant_size_upload_parts: bool,
    pub list_is_lexically_ordered: bool,
    pub store_prefix: String,
    /* private fields */
}

Wraps ObjectStore

Fields

inner: Arc<dyn OSObjectStore>

use_constant_size_upload_parts: bool

Whether to use constant size upload parts for multipart uploads. This is only necessary for Cloudflare R2.

list_is_lexically_ordered: bool

Whether we can assume that the list of files is lexically ordered. This is true for object stores, but not for local filesystems.

store_prefix: String

The datastore prefix that uniquely identifies this object store. It encodes information which usually cannot be found in the URL such as Azure account name. The prefix plus the path uniquely identifies any object inside the store.

Implementations

impl ObjectStore

pub async fn read_dir_page( &self, dir: impl Into<Path>, options: ReadDirOptions, ) -> Result<PaginatedListResult>

One page of the immediate children of dir, one directory level deep.

On backends with a paginated list API — S3, GCS and Azure — the resume position and the page size are pushed into the list request, so the page costs what the page holds. Elsewhere the directory is listed in full and paged locally, which is correct but no cheaper than Self::read_dir.

Child directories come back as ListResult::common_prefixes and child objects as ListResult::objects, the same split Self::list_with_delimiter returns.

One page is one request, so a page can hold fewer children than limit asked for and still be followed by more: with a delimiter a backend spends its page budget on keys it collapses away, and it has a cap of its own besides. Walk until PaginatedListResult::page_token is None rather than until a page comes back short.

let mut tables = Vec::new();
let mut page_token = None;
loop {
    let page = store
        .read_dir_page("my_db", ReadDirOptions { page_token, limit: Some(10) })
        .await?;
    // A table is a directory, so a loose object that happens to be named like one is not
    // a table.
    tables.extend(page.result.common_prefixes.iter().filter_map(|table| {
        Some(table.filename()?.strip_suffix(".lance")?.to_string())
    }));
    page_token = page.page_token;
    if page_token.is_none() || tables.len() >= 10 {
        break;
    }
}

impl ObjectStore

pub async fn from_uri(uri: &str) -> Result<(Arc<Self>, Path)>

Parse from a string URI.

Returns the ObjectStore instance and the absolute path to the object.

This uses the default ObjectStoreRegistry to find the object store. To allow for potential re-use of object store instances, it’s recommended to create a shared ObjectStoreRegistry and pass that to Self::from_uri_and_params.

pub async fn from_uri_and_params( registry: Arc<ObjectStoreRegistry>, uri: &str, params: &ObjectStoreParams, ) -> Result<(Arc<Self>, Path)>

Parse from a string URI.

Returns the ObjectStore instance and the absolute path to the object.

pub fn extract_path_from_uri( registry: Arc<ObjectStoreRegistry>, uri: &str, ) -> Result<Path>

Extract the path component from a URI without initializing the object store.

This is a synchronous operation that only parses the URI and extracts the path, without creating or initializing any object store instance.

Arguments

• registry - The object store registry to get the provider
• uri - The URI to extract the path from

Returns

The extracted path component

pub fn from_path(str_path: &str) -> Result<(Arc<Self>, Path)>

👎Deprecated:

Use from_uri instead

pub fn local() -> Self

Local object store.

pub fn memory() -> Self

Create a in-memory object store directly for testing.

pub fn is_local(&self) -> bool

Returns true if the object store pointed to a local file system.

pub fn has_direct_local_paths(&self) -> bool

Returns true when object paths directly encode absolute local filesystem paths.

Local stores rooted below the filesystem root, such as UNC-backed stores, use their inner object-store implementation instead of direct filesystem access.

pub fn is_cloud(&self) -> bool

pub fn prefers_lite_scheduler(&self) -> bool

Whether this object store prefers the lite scheduler.

The lite scheduler is designed for backends like io_uring where tasks should only be polled when the consumer polls them.

pub fn scheme(&self) -> &str

pub fn block_size(&self) -> usize

pub fn max_iop_size(&self) -> u64

pub fn io_parallelism(&self) -> usize

The amount of parallelism to use for I/O operations.

Honors the LANCE_IO_THREADS override when set, otherwise the store’s configured value. Always at least 1: callers feed this straight into buffered / buffer_unordered, and a window of 0 makes those streams never poll their input — e.g. a metadata-only count_rows would hang rather than return.

pub fn io_tracker(&self) -> &IOTracker

Get the IO tracker for this object store

The IO tracker can be used to get statistics about read/write operations performed on this object store.

pub fn io_stats_snapshot(&self) -> IoStats

Get a snapshot of current IO statistics without resetting counters

Returns the current IO statistics without modifying the internal state. Use this when you need to check stats without resetting them.

pub fn io_stats_incremental(&self) -> IoStats

Get incremental IO statistics since the last call to this method

Returns the accumulated statistics since the last call and resets the counters to zero. This is useful for tracking IO operations between different stages of processing.

pub fn apply_wrapper(&mut self, wrapper: &dyn WrappingObjectStore)

Apply a WrappingObjectStore to both inner and paginated_lister together.

Keeps both halves in sync: a wrapper returning None from WrappingObjectStore::wrap_paginated clears the lister so that Self::read_dir_page falls back through the (already-wrapped) inner.

pub async fn open(&self, path: &Path) -> Result<Box<dyn Reader>>

Open a file for path.

Parameters

• path: Absolute path to the file.

pub async fn open_with_size( &self, path: &Path, known_size: usize, ) -> Result<Box<dyn Reader>>

Open a reader for a file with known size.

This size may either have been retrieved from a list operation or cached metadata. By passing in the known size, we can skip a HEAD / metadata call.

pub async fn create_local_writer(path: &Path) -> Result<ObjectWriter>

Create an ObjectWriter from local std::path::Path

pub async fn open_local(path: &Path) -> Result<Box<dyn Reader>>

Open an Reader from local std::path::Path

pub async fn create(&self, path: &Path) -> Result<Box<dyn Writer>>

Create a new file.

pub async fn put(&self, path: &Path, content: &[u8]) -> Result<WriteResult>

A helper function to create a file and write content to it.

pub async fn put_if_absent( &self, path: &Path, content: PutPayload, ) -> Result<()>

Atomically creates an object without replacing an existing object.

Local stores publish a uniquely named staging object with a conditional rename. Other stores use their conditional create operation. Tencent COS is rejected because it can silently ignore conditional create requests.

Returns object_store::Error::NotSupported without writing when the backend cannot reliably provide put-if-absent semantics.

pub async fn delete(&self, path: &Path) -> Result<()>

pub async fn copy(&self, from: &Path, to: &Path) -> Result<()>

pub async fn copy_bulk( &self, source_path: &Path, destination_store: &Self, destination_path: &Path, ) -> Result<WriteResult>

Copy an object using the policy for bulk file movement.

Streaming is the default because it works across object stores and does not require provider-native copy support. Setting LANCE_IO_SERVER_SIDE_COPY_ENABLED to a truthy value opts same-store copies into Self::copy. Cross-store and local copies continue to use Self::copy_via_stream.

source
    .copy_bulk(
        &Path::from("staging/index.lance"),
        destination,
        &Path::from("index.lance"),
    )
    .await?;

pub async fn copy_via_stream( &self, source_path: &Path, destination_store: &Self, destination_path: &Path, ) -> Result<WriteResult>

Copy an object by streaming its bytes through Lance’s multipart-aware writer.

Unlike Self::copy, this never delegates to a provider-native server-side copy. The source and destination may use different object stores. The copy succeeds only after the byte count reported by the writer and a destination metadata lookup both match the source size.

source
    .copy_via_stream(
        &Path::from("staging/index.lance"),
        destination,
        &Path::from("index.lance"),
    )
    .await?;

pub async fn read_dir(&self, dir_path: impl Into<Path>) -> Result<Vec<String>>

Read a directory (start from base directory) and returns all sub-paths in the directory.

This enumerates the whole prefix before it returns, however many children it holds. Use Self::read_dir_page to page through a directory instead.

pub async fn list_with_delimiter( &self, prefix: Option<&Path>, ) -> Result<ListResult>

Non-recursive, path-segment delimited list of a single directory level.

Unlike Self::list, which recurses into the entire subtree, this returns only the immediate children of prefix: the child “directories” as ListResult::common_prefixes and the direct child files as ListResult::objects.

pub fn list( &self, path: Option<Path>, ) -> Pin<Box<dyn Stream<Item = Result<ObjectMeta>> + Send>>

pub fn read_dir_all<'a, 'b>( &'a self, dir_path: impl Into<&'b Path> + Send, unmodified_since: Option<DateTime<Utc>>, ) -> BoxStream<'a, Result<ObjectMeta>>

Read all files (start from base directory) recursively

unmodified_since can be specified to only return files that have not been modified since the given time.

pub async fn remove_dir_all(&self, dir_path: impl Into<Path>) -> Result<()>

Remove a directory recursively.

pub async fn remove_empty_dirs( &self, root_path: impl Into<Path>, retained_dirs: HashSet<Path>, verified_dirs: HashSet<Path>, unmodified_since: Option<DateTime<Utc>>, ) -> Result<()>

Remove eligible materialized empty directories below a local root.

This is a no-op for object stores, which do not materialize directories. Traversal does not follow symbolic links. Directories in retained_dirs and their descendants are preserved. Other directories are removed only if they are empty and either appear in verified_dirs or predate unmodified_since. Passing None for unmodified_since disables the age check.

store
    .remove_empty_dirs(
        "dataset/_indices",
        HashSet::new(),
        HashSet::new(),
        Some(Utc::now()),
    )
    .await?;

pub fn remove_stream<'a>( &'a self, locations: BoxStream<'a, Result<Path>>, ) -> BoxStream<'a, Result<Path>>

pub async fn exists(&self, path: &Path) -> Result<bool>

Check a file exists.

pub async fn size(&self, path: &Path) -> Result<u64>

Get file size.

pub async fn read_one_all(&self, path: &Path) -> Result<Bytes>

Convenience function to open a reader and read all the bytes

pub async fn read_one_range( &self, path: &Path, range: Range<usize>, ) -> Result<Bytes>

Convenience function open a reader and make a single request

If you will be making multiple requests to the path it is more efficient to call Self::open and then call Reader::get_range multiple times.

impl ObjectStore

pub fn new( store: Arc<DynObjectStore>, location: Url, block_size: Option<usize>, wrapper: Option<Arc<dyn WrappingObjectStore>>, use_constant_size_upload_parts: bool, list_is_lexically_ordered: bool, io_parallelism: usize, download_retry_count: usize, storage_options: Option<&HashMap<String, String>>, ) -> Self

Trait Implementations

impl Clone for ObjectStore

fn clone(&self) -> ObjectStore

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ObjectStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DeepSizeOf for ObjectStore

fn deep_size_of_children(&self, context: &mut Context) -> usize

fn deep_size_of(&self) -> usize

impl Display for ObjectStore

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.