zarrs is Rust library for the Zarr storage format for multidimensional arrays and metadata. It supports:
- Zarr V3, and
- A V3 compatible subset of Zarr V2.
A changelog can be found here. Correctness issues with past versions are detailed here.
Developed at the Department of Materials Physics, Australian National University, Canberra, Australia.
Getting Started
- Review the implementation status, array support, and storage support.
- View the examples and the example below.
- Read the documentation. [
array::Array] is a good place to start. - Check out
zarrsecosystem.
Implementation Status
zarrs has first-class Zarr V3 support and additionally supports a compatible subset of Zarr V2 data that:
- can be converted to V3 with only a metadata change, and
- uses array metadata that is recognised and supported for encoding/decoding.
An existing V2 or V3 array can be opened with Array::open.
A new array can be created from V2 or V3 metadata with Array::new_with_metadata.
The ArrayBuilder only supports V3 array creation.
zarrs supports forward conversion of Zarr V2 data to V3.
See "Metadata Convert Version" and "Metadata Erase Version" for information about manipulating the version of array/group metadata.
Array Support
Storage Support
zarrs supports stores (filesystem, HTTP, S3, etc.) via crates implementing the [zarrs_storage] API.
A huge range of storage backends are supported via the opendal and object_store crates.
The documentation for the [zarrs_opendal] and [zarrs_object_store] crates includes version compatibility matrices with zarrs and the associated storage backends.
These backends provide more feature complete HTTP stores than [zarrs_http].
Asynchronous stores can be used in a synchronous context with the AsyncToSyncStorageAdapter.
Examples
use std::{path::PathBuf, sync::Arc};
use zarrs::group::GroupBuilder; use zarrs::array::{ArrayBuilder, DataType, FillValue, ZARR_NAN_F32};
#[cfg(feature = "gzip")]
use zarrs::array::codec::GzipCodec; // requires gzip feature use zarrs::array_subset::ArraySubset; use zarrs::storage::ReadableWritableListableStorage; use zarrs_filesystem::FilesystemStore;
// Create a filesystem store let store_path: PathBuf = "/path/to/hierarchy.zarr".into();
let store_path: PathBuf = "tests/data/array_write_read.zarr".into();
let store: ReadableWritableListableStorage = Arc::new(FilesystemStore::new(&store_path)?);
let store = Arc::new(zarrs::storage::store::MemoryStore::new());
// Write the root group metadata GroupBuilder::new() .build(store.clone(), "/")? // .attributes(...) .store_metadata()?;
// Create a new V3 array using the array builder let array = ArrayBuilder::new( vec![3, 4], // array shape DataType::Float32, vec![2, 2].try_into()?, // regular chunk shape (non-zero elements) FillValue::from(ZARR_NAN_F32), ) .bytes_to_bytes_codecs(vec![
#[cfg(feature = "gzip")]
Arc::new(GzipCodec::new(5)?), ]) .dimension_names(["y", "x"].into()) .attributes(serde_json::json!({"Zarr V3": "is great"}).as_object().unwrap().clone()) .build(store.clone(), "/array")?; // /path/to/hierarchy.zarr/array
// Store the array metadata array.store_metadata()?; println!("{}", serde_json::to_string_pretty(array.metadata())?); // { // "zarr_format": 3, // "node_type": "array", // ... // }
// Perform some operations on the chunks array.store_chunk_elements::( &[0, 1], // chunk index &[0.2, 0.3, 1.2, 1.3] )?; array.store_array_subset_ndarray::<f32, _>( &[1, 1], // array index ndarray::array![[-1.1, -1.2], [-2.1, -2.2]] )?; array.erase_chunk(&[1, 1])?;
// Retrieve all array elements as an ndarray let array_subset_all = ArraySubset::new_with_shape(array.shape().to_vec()); let array_ndarray = array.retrieve_array_subset_ndarray::(&array_subset_all)?; println!("{array_ndarray:4}"); // [[ NaN, NaN, 0.2, 0.3], // [ NaN, -1.1, -1.2, 1.3], // [ NaN, -2.1, NaN, NaN]]
Ok::<(), Box>(())
Examples can be run with `cargo run --example <EXAMPLE_NAME>`.
- Add `-- --usage-log` to see storage API calls during example execution.
- Some examples require non-default features, which can be enabled with `--all-features` or `--features <FEATURES>`.
#### Sync API Examples