Crate pyo3_object_store

Expand description

§pyo3-object_store

Integration between object_store and pyo3.

This provides Python builder classes so that Python users can easily create Arc<dyn ObjectStore> instances, which can then be used in pure-Rust code.

§Usage

Register the builders.

#[pymodule]
fn python_module(py: Python, m: &Bound<PyModule>) -> PyResult<()> {
    pyo3_object_store::register_store_module(py, m, "python_module", "store")?;
    pyo3_object_store::register_exceptions_module(py, m, "python_module", "exceptions")?;
}

This exports the underlying Python classes from your own Rust-Python library.

Refer to register_store_module and register_exceptions_module for more information.

Accept PyObjectStore as a parameter in your function exported to Python. Its into_dyn method (or Into impl) gives you an Arc<dyn ObjectStore>.
```
#[pyfunction]
pub fn use_object_store(store: PyObjectStore) {
    let store: Arc<dyn ObjectStore> = store.into_dyn();
}
```
You can also accept AnyObjectStore as a parameter, which wraps PyObjectStore and PyExternalObjectStore. This allows you to seamlessly recreate ObjectStore instances that users pass in from other Python libraries (like obstore) that themselves export pyo3-object_store builders.

Note however that due to lack of ABI stability, ObjectStore instances will be recreated, and so there will be no connection pooling across the external store.

§Example

The obstore Python library gives a full real-world example of using pyo3-object_store, exporting a Python API that mimics the Rust ObjectStore API.

§Feature flags

external-store-warning (enabled by default): Emit a user warning when constructing a PyExternalObjectStore (or AnyObjectStore::PyExternalObjectStore), to inform users that there may be performance implications due to lack of connection pooling across separately-compiled Python libraries. Disable this feature if you don’t want the warning.

§ABI stability

It’s not currently possible to share a #[pyclass] across multiple Python libraries, except in special cases where the underlying data has a stable ABI.

As object_store does not currently have a stable ABI, we can’t share PyObjectStore instances across multiple separately-compiled Python libraries.

We have two ways to get around this:

Export your own Python classes so that users can construct ObjectStore instances that were compiled with your library. See register_store_module.
Accept AnyObjectStore or PyExternalObjectStore as a parameter, which allows for seamlessly reconstructing stores from an external Python library, like obstore. This has some overhead and removes any possibility of connection pooling across the two instances.

Note about not being able to use these across Python packages. It has to be used with the exported classes from your own library.

§Python Type hints

We don’t yet have a great solution here for reusing the store builder type hints in your own library. Type hints are shipped with the cargo dependency. Or, you can use a submodule on the obstore repo. See async-tiff for an example.

§Version compatibility

pyo3-object_store	pyo3	object_store
0.1.x	0.23	0.12
0.2.x	0.24	0.12
0.3.x	0.23 :warning:	0.11 :warning:
0.4.x	0.24	0.11 :warning:
0.5.x	0.25	0.12
0.6.x	0.26	0.12
0.7.x	0.27	0.12

Note that 0.3.x and 0.4.x are compatibility releases to use pyo3-object_store with older versions of pyo3 and object_store.

Structs§

MaybePrefixedStore: Store wrapper that applies a constant prefix to all paths handled by the store.
PyAzureStore: A Python-facing wrapper around a MicrosoftAzure.
PyClientConfigKey: A wrapper around ClientConfigKey that implements FromPyObject.
PyClientOptions: A wrapper around ClientOptions that implements FromPyObject.
PyExternalObjectStore: A wrapper around a Rust ObjectStore instance that will extract and recreate an ObjectStore instance out of a Python object.
PyGCSStore: A Python-facing wrapper around a GoogleCloudStorage.
PyHttpStore: A Python-facing wrapper around a HttpStore.
PyLocalStore: A Python-facing wrapper around a LocalFileSystem.
PyMemoryStore: A Python-facing wrapper around an InMemory.
PyObjectStore: A wrapper around a Rust ObjectStore instance that allows any rust-native implementation of ObjectStore.
PyPath: A Python-facing wrapper around a Path.
PyS3Store: A Python-facing wrapper around an AmazonS3.
PyUrl: A wrapper around url::Url that implements FromPyObject.

Enums§

AnyObjectStore: A convenience wrapper around native and external ObjectStore instances.
PyObjectStoreError: The Error variants returned by this crate.

Functions§

from_url: Simple construction of stores by url.
register_exceptions_module: Export exceptions as a submodule named exceptions within the given parent module
register_store_module: Export the default Python API as a submodule named store within the given parent module

Type Aliases§

PyObjectStoreResult: A type wrapper around Result<T, PyObjectStoreError>.