pyo3-object_store

Integration between object_store and pyo3.

This provides Python builder classes so that Python users can easily create [Arc<dyn ObjectStore>][object_store::ObjectStore] instances, which can then be used in pure-Rust code.

Usage

1. Register the builders.

   #[pymodule]
   fn python_module(py: Python, m: &Bound<PyModule>) -> PyResult<()> {
       pyo3_object_store::register_store_module(py, m, "python_module", "store")?;
       pyo3_object_store::register_exceptions_module(py, m, "python_module", "exceptions")?;
   }
   

   This exports the underlying Python classes from your own Rust-Python library.

   Refer to [register_store_module] and [register_exceptions_module] for more information.

2. Accept [PyObjectStore] as a parameter in your function exported to Python. Its [into_dyn][PyObjectStore::into_dyn] method (or Into impl) gives you an [Arc<dyn ObjectStore>][object_store::ObjectStore].

   #[pyfunction]
   pub fn use_object_store(store: PyObjectStore) {
       let store: Arc<dyn ObjectStore> = store.into_dyn();
   }
   

   You can also accept [AnyObjectStore] as a parameter, which wraps [PyObjectStore] and [PyExternalObjectStore]. This allows you to seamlessly recreate ObjectStore instances that users pass in from other Python libraries (like obstore) that themselves export pyo3-object_store builders.

   Note however that due to lack of ABI stability, ObjectStore instances will be recreated, and so there will be no connection pooling across the external store.

Example

The obstore Python library gives a full real-world example of using pyo3-object_store, exporting a Python API that mimics the Rust [ObjectStore][object_store::ObjectStore] API.

Feature flags

• external-store-warning (enabled by default): Emit a user warning when constructing a PyExternalObjectStore (or AnyObjectStore::PyExternalObjectStore), to inform users that there may be performance implications due to lack of connection pooling across separately-compiled Python libraries. Disable this feature if you don't want the warning.

ABI stability

It's not currently possible to share a #[pyclass] across multiple Python libraries, except in special cases where the underlying data has a stable ABI.

As object_store does not currently have a stable ABI, we can't share PyObjectStore instances across multiple separately-compiled Python libraries.

We have two ways to get around this:

• Export your own Python classes so that users can construct ObjectStore instances that were compiled with your library. See [register_store_module].
• Accept [AnyObjectStore] or [PyExternalObjectStore] as a parameter, which allows for seamlessly reconstructing stores from an external Python library, like obstore. This has some overhead and removes any possibility of connection pooling across the two instances.

Note about not being able to use these across Python packages. It has to be used with the exported classes from your own library.

Python Type hints

We don't yet have a great solution here for reusing the store builder type hints in your own library. Type hints are shipped with the cargo dependency. Or, you can use a submodule on the obstore repo. See async-tiff for an example.

Version compatibility

Note that 0.3.x and 0.4.x are compatibility releases to use pyo3-object_store with older versions of pyo3 and object_store.