Struct ObjectStoreMirror 

pub struct ObjectStoreMirror { /* private fields */ }

An ObjectStore wrapper that keeps whole objects in a local directory.

The remote store holds the authoritative objects. Local copies contain the same bytes, without splitting objects into blocks or changing their format. A MirrorPolicy chooses which objects to mirror. The mirror does not impose a size limit or automatically evict objects to reclaim disk space.

Read and write routes

A MirrorPolicy selects how each request uses the remote store and local copies:

• GET and HEAD use ReadRoute.
• PUT uses WriteRoute.
• Multipart uploads use MultipartRoute.

The Observe routes pass the whole object to MirrorPolicy::observe. Multipart uploads do not support observation.

Policies manage local copies through MirrorHandle:

• fetch waits for a local copy.
• prefetch schedules a background download.
• evict removes a local copy.

Fetching an existing local copy does not refresh it. Use ReadRoute::Refetch to replace it.

Downloads, mirrored writes, deletes, and evictions coordinate per path. A later download or mirrored write can cancel an eviction that has not started.

Listings always use the remote store. Deletes remove the remote object and its local copy. Copy and rename operations return MirrorError::Unsupported.

Object lifetime and local integrity

Policies must only mirror paths that are never overwritten. The mirror does not track remote updates to existing objects. Route mutable objects through Remote or Observe.

Local copies are disposable. The default StdVfs flushes writes without syncing them to disk. Local reads can return corrupted bytes without an error, including after a machine crash. Startup checks metadata, filenames, and file sizes, but does not check object contents.

Callers must detect corruption and signal their policy to route a retry through ReadRoute::Refetch. Repeating a ReadRoute::Local read does not repair the local copy. Callers must also check the refetched data for corruption and limit the number of retries.

Startup and background tasks

Call ObjectStoreMirrorBuilder::build inside a Tokio runtime. It creates the local directory, acquires its exclusive lock, and recovers complete local copies. Use a dedicated directory for each mirror and remote store.

The builder starts MirrorPolicy::run and a periodic remote scan. The scan removes local copies of objects deleted remotely. It does not download new objects or refresh existing copies. The scan requires strongly consistent remote reads, writes, deletes, and listings. If the remote store lacks these guarantees, disable the scan with with_remote_scan_interval(None).

The defaults allow eight concurrent downloads and a remote scan every ten minutes. Downloads and remote scan listings retry transient errors without a limit by default. Use ObjectStoreMirrorBuilder::with_max_retries to bound these retries.

Dropping the last Arc<ObjectStoreMirror> cancels the background tasks. Cloned MirrorHandle values keep the local state and directory lock alive until those handles are also dropped.

Errors

Mirror errors travel inside object_store::Error. Use MirrorError::find to recover the specific error from its source chain. MirrorError::WriteCommitted means that the remote write succeeded, but local installation or the observation hook failed. Account for that remote write before deciding whether to retry it.

Example

This policy mirrors every write and serves every read locally. It assumes that callers write each path once. Existing remote objects need an explicit MirrorHandle::fetch before a local read.

use std::sync::Arc;

use async_trait::async_trait;
use object_store::memory::InMemory;
use object_store::path::Path;
use object_store::{
    GetOptions, ObjectStore, ObjectStoreExt, PutMultipartOptions, PutOptions,
};
use slatedb_mirror::{MirrorPolicy, MultipartRoute, ObjectStoreMirror, ReadRoute, WriteRoute};

struct ImmutableObjects;

#[async_trait]
impl MirrorPolicy for ImmutableObjects {
    fn read_route(&self, _: &Path, _: &GetOptions) -> object_store::Result<ReadRoute> {
        Ok(ReadRoute::Local)
    }

    fn put_route(&self, _: &Path, _: &PutOptions) -> object_store::Result<WriteRoute> {
        Ok(WriteRoute::Mirror)
    }

    fn put_multipart_route(
        &self,
        _: &Path,
        _: &PutMultipartOptions,
    ) -> object_store::Result<MultipartRoute> {
        Ok(MultipartRoute::Mirror)
    }
}

let local_dir = tempfile::tempdir()?;
let remote: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
let mirror = ObjectStoreMirror::builder(
    local_dir.path(),
    remote,
    Arc::new(ImmutableObjects),
)
.build()
.await?;

let path = Path::from("objects/first.bin");
mirror.put(&path, "hello".into()).await?;
assert!(mirror.handle().contains(&path));
let bytes = mirror.get(&path).await?.bytes().await?;
assert_eq!(bytes.as_ref(), b"hello");

Implementations

impl ObjectStoreMirror

pub fn builder( local_dir: impl Into<PathBuf>, object_store: Arc<dyn ObjectStore>, policy: Arc<dyn MirrorPolicy>, ) -> ObjectStoreMirrorBuilder

pub fn handle(&self) -> MirrorHandle

A handle to this mirror, like the one the policy gets.

Trait Implementations

impl Debug for ObjectStoreMirror

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for ObjectStoreMirror

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for ObjectStoreMirror

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl ObjectStore for ObjectStoreMirror

fn put_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, payload: PutPayload, opts: PutOptions, ) -> Pin<Box<dyn Future<Output = Result<PutResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Save the provided payload to location with the given options

fn put_multipart_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, opts: PutMultipartOptions, ) -> Pin<Box<dyn Future<Output = Result<Box<dyn MultipartUpload>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a multipart upload with options

fn get_opts<'life0, 'life1, 'async_trait>( &'life0 self, location: &'life1 Path, options: GetOptions, ) -> Pin<Box<dyn Future<Output = Result<GetResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Perform a get request with options

fn delete_stream( &self, locations: BoxStream<'static, Result<Path>>, ) -> BoxStream<'static, Result<Path>>

Delete all the objects at the specified locations

fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>>

List all the objects with the given prefix.

fn list_with_offset( &self, prefix: Option<&Path>, offset: &Path, ) -> BoxStream<'static, Result<ObjectMeta>>

List all the objects with the given prefix and a location greater than offset

fn list_with_delimiter<'life0, 'life1, 'async_trait>( &'life0 self, prefix: Option<&'life1 Path>, ) -> Pin<Box<dyn Future<Output = Result<ListResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

List objects with the given prefix and an implementation specific delimiter. Returns common prefixes (directories) in addition to object metadata.

fn copy_opts<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, _from: &'life1 Path, _to: &'life2 Path, _options: CopyOptions, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Copy an object from one path to another in the same object store.

fn rename_opts<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, _from: &'life1 Path, _to: &'life2 Path, _options: RenameOptions, ) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Move an object from one path to another in the same object store.

fn get_ranges<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, location: &'life1 Path, ranges: &'life2 [Range<u64>], ) -> Pin<Box<dyn Future<Output = Result<Vec<Bytes>, Error>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, Self: 'async_trait,

Return the bytes that are stored at the specified location in the given byte ranges