Crate zarrs

Expand description

zarrs is Rust library for the Zarr storage format for multidimensional arrays and metadata.

If you are a Python user, check out zarrs-python. It includes a high-performance codec pipeline for the reference zarr-python implementation.

zarrs supports Zarr V3 and a V3 compatible subset of Zarr V2. It is fully up-to-date and conformant with the Zarr 3.1 specification with support for:

all core extensions (data types, codecs, chunk grids, chunk key encodings, storage transformers),
all accepted Zarr Enhancement Proposals (ZEPs) and several draft ZEPs:
- ZEP 0003: Variable chunking
- ZEP 0007: Strings
- ZEP 0009: Zarr Extension Naming
various registered extensions from zarr-developers/zarr-extensions/,
experimental codecs and data types intended for future registration, and
user-defined custom extensions and stores.

A changelog can be found here. Correctness issues with past versions are detailed here.

Developed at the Department of Materials Physics, Australian National University, Canberra, Australia.

§Getting Started

Review the implementation status which summarises zarr version support, array support (codecs, data types, etc.) and storage support.
Read The zarrs Book.
View the examples and the example below.
Read the documentation.
Check out the zarrs ecosystem.

§Implementation Status

§Zarr Version Support

zarrs has first-class Zarr V3 support and additionally supports a compatible subset of Zarr V2 data that:

can be converted to V3 with only a metadata change, and
uses array metadata that is recognised and supported for encoding/decoding.

zarrs supports forward conversion from Zarr V2 to V3. See “Converting Zarr V2 to V3” in The zarrs Book, or try the zarrs_reencode CLI tool.

§Array Support

Data Types

`DataType`	V3 `name`	V2 `dtype`	`ElementOwned` / `Element`	Feature Flag
`Bool`	bool	\|b1	`bool`
`Int2`	int2		`i8`
`Int4`	int4		`i8`
`Int8`	int8	\|i1	`i8`
`Int16`	int16	>i2 <i2	`i16`
`Int32`	int32	>i4 <i4	`i32`
`Int64`	int64	>i8 <i8	`i64`
`UInt2`	uint2		`u8`
`UInt4`	uint4		`u8`
`UInt8`	uint8	\|u1	`u8`
`UInt16`	uint16	>u2 <u2	`u16`
`UInt32`	uint32	>u4 <u4	`u32`
`UInt64`	uint64	>u8 <u8	`u64`
`Float4E2M1FN`	float4_e2m1fn
`Float6E2M3FN`	float6_e2m3fn
`Float6E3M2FN`	float6_e3m2fn
`Float8E3M4`	float8_e3m4
`Float8E4M3`	float8_e4m3
`Float8E4M3B11FNUZ`	float8_e4m3b11fnuz
`Float8E4M3FNUZ`	float8_e4m3fnuz
`Float8E5M2`	float8_e5m2
`Float8E5M2FNUZ`	float8_e5m2fnuz
`Float8E8M0FNU`	float8_e8m0fnu
`BFloat16`	bfloat16		`half::bf16`
`Float16`	float16	>f2 <f2	`half::f16`
`Float32`	float32	>f4 <f4	`f32`
`Float64`	float64	>f8 <f8	`f64`
`ComplexBFloat16`	complex_bfloat16		`Complex<half::bf16>`
`ComplexFloat16`	complex_float16		`Complex<half::f16>`
`ComplexFloat32`	complex_float32		`Complex<f32>`
`ComplexFloat64`	complex_float64		`Complex<f64>`
`Complex64`	complex64	>c8 <c8	`Complex<f32>`
`Complex128`	complex128	>c16 <c16	`Complex<f64>`
`RawBits`	r		`[u8; N]` / `&[u8; N]`
`String`	string	\|O	`String` / `&str`
`Bytes`	bytes binary	\|VX	`Vec<u8>` / `&[u8]`
`NumpyDateTime64`	numpy.datetime64		`i64` `chrono::DateTime<Utc>` `jiff::Timestamp`	chrono jiff
`NumpyTimeDelta64`	numpy.timedelta64		`i64` `chrono::TimeDelta` `jiff::SignedDuration`	chrono jiff

Codecs

Codec Type	Default codec `name`	Specification	Feature Flag*
Array to Array	`transpose`	Zarr V3.0 Transpose	transpose
	`numcodecs.fixedscaleoffset`	Experimental
	`numcodecs.bitround`†	Experimental	bitround
	`zarrs.squeeze`	Experimental
Array to Bytes	`bytes`	Zarr V3.0 Bytes
	`sharding_indexed`	Zarr V3.0 Sharding	sharding
	`vlen-array`	Experimental
	`vlen-bytes`	zarr-extensions/codecs/vlen-bytes
	`vlen-utf8`	zarr-extensions/codecs/vlen-utf8
	`numcodecs.pcodec`	Experimental	pcodec
	`numcodecs.zfpy`	Experimental	zfp
	`packbits`	zarr-extensions/codecs/packbits
	`zarrs.vlen`	Experimental
	`zarrs.vlen_v2`	Experimental
	`zfp`	zarr-extensions/codecs/zfp	zfp
Bytes to Bytes	`blosc`	Zarr V3.0 Blosc	blosc
	`crc32c`	Zarr V3.0 CRC32C	crc32c
	`gzip`	Zarr V3.0 Gzip	gzip
	`zstd`	zarr-extensions/codecs/zstd	zstd
	`numcodecs.bz2`	Experimental	bz2
	`numcodecs.fletcher32`	Experimental	fletcher32
	`numcodecs.shuffle`	Experimental
	`numcodecs.zlib`	Experimental	zlib
	`zarrs.gdeflate`	Experimental	gdeflate

^{* Bolded feature flags are part of the default set of features.} ^{† numcodecs.bitround supports additional data types not supported by zarr-python/numcodecs}

Codecs have three potential statuses:

Core: These are defined in the Zarr V3 specification and are fully supported.
Registered: These are specified at https://github.com/zarr-developers/zarr-extensions/ and are fully supported unless otherwise indicated.
Experimental: These are recommended for evaluation only.
- These codecs may have no formal specification or are pending registration at https://github.com/zarr-developers/zarr-extensions/.
- These codecs may change in future releases without maintaining backwards compatibility.

Codec names and aliases are configurable with Config::codec_aliases_v3_mut and Config::codec_aliases_v2_mut. zarrs will persist codec names if opening an existing array of creating an array from metadata.

zarrs supports arrays created with zarr-python 3.x.x with various numcodecs.zarr3 codecs. However, arrays must be written with numcodecs 0.15.1+.

Chunk Grids

Chunk Grid	ZEP	V3	V2	Feature Flag
regular	ZEP0001	✓	✓
rectangular (experimental)	ZEP0003 (draft)	✓

Chunk Key Encodings

Chunk Key Encoding	ZEP	V3	V2	Feature Flag
default	ZEP0001	✓
v2	ZEP0001	✓	✓

Storage Transformers

Zarr V3 does not currently define any storage transformers.

§Storage Support

zarrs supports a huge range of stores (including custom stores) via the zarrs_storage API.

Stores

Store/Storage Adapter	ZEP	Read	Write	List	Sync	Async	Crate
MemoryStore		✓	✓	✓	✓		zarrs_storage^†
FilesystemStore	0001	✓	✓	✓	✓		zarrs_filesystem^‡
OpendalStore		✓*	✓*	✓*	✓		zarrs_opendal
AsyncOpendalStore		✓*	✓*	✓*		✓	zarrs_opendal
AsyncObjectStore		✓*	✓*	✓*		✓	zarrs_object_store
AsyncIcechunkStore		✓*	✓*	✓*		✓	zarrs_icechunk
HTTPStore		✓			✓		zarrs_http
AsyncToSyncStorageAdapter		✓	✓	✓	✓	✓	zarrs_storage^†
UsageLogStorageAdapter		✓	✓	✓	✓	✓	zarrs_storage^†
PerformanceMetricsStorageAdapter		✓	✓	✓	✓	✓	zarrs_storage^†
ZipStorageAdapter		✓		✓	✓		zarrs_zip

^{† Re-exported in the zarrs::storage module.}
^{‡ Re-exported as the zarrs::filesystem module.}
^{* Support depends on the underlying store.}

The opendal and object_store crates are popular Rust storage backends that are fully supported via zarrs_opendal and zarrs_object_store. These backends provide more feature complete HTTP stores than zarrs_http.

zarrs_icechunk implements the Icechunk transactional storage engine, a storage specification for Zarr that supports object_store stores.

The AsyncToSyncStorageAdapter enables some async stores to be used in a sync context.

§Examples

§Create and Read a Zarr Hierarchy

use zarrs::group::GroupBuilder;
use zarrs::array::{ArrayBuilder, DataType, FillValue, ZARR_NAN_F32};
use zarrs::array::codec::GzipCodec; // requires gzip feature
use zarrs::array_subset::ArraySubset;
use zarrs::storage::ReadableWritableListableStorage;
use zarrs::filesystem::FilesystemStore; // requires filesystem feature

// Create a filesystem store
let store_path: PathBuf = "/path/to/hierarchy.zarr".into();
let store: ReadableWritableListableStorage =
    Arc::new(FilesystemStore::new(&store_path)?);

// Write the root group metadata
GroupBuilder::new()
    .build(store.clone(), "/")?
    // .attributes(...)
    .store_metadata()?;

// Create a new V3 array using the array builder
let array = ArrayBuilder::new(
    vec![3, 4], // array shape
    DataType::Float32,
    vec![2, 2].try_into()?, // regular chunk shape (non-zero elements)
    FillValue::from(ZARR_NAN_F32),
)
.bytes_to_bytes_codecs(vec![
    Arc::new(GzipCodec::new(5)?),
])
.dimension_names(["y", "x"].into())
.attributes(serde_json::json!({"Zarr V3": "is great"}).as_object().unwrap().clone())
.build(store.clone(), "/array")?; // /path/to/hierarchy.zarr/array

// Store the array metadata
array.store_metadata()?;
println!("{}", serde_json::to_string_pretty(array.metadata())?);
// {
//     "zarr_format": 3,
//     "node_type": "array",
//     ...
// }

// Perform some operations on the chunks
array.store_chunk_elements::<f32>(
    &[0, 1], // chunk index
    &[0.2, 0.3, 1.2, 1.3]
)?;
array.store_array_subset_ndarray::<f32, _>(
    &[1, 1], // array index (start of subset)
    ndarray::array![[-1.1, -1.2], [-2.1, -2.2]]
)?;
array.erase_chunk(&[1, 1])?;

// Retrieve all array elements as an ndarray
let array_ndarray = array.retrieve_array_subset_ndarray::<f32>(&array.subset_all())?;
println!("{array_ndarray:4}");
// [[ NaN,  NaN,  0.2,  0.3],
//  [ NaN, -1.1, -1.2,  1.3],
//  [ NaN, -2.1,  NaN,  NaN]]

§More examples

Various examples can be found in the examples directory that demonstrate:

creating and manipulating zarr hierarchies with various stores (sync and async), codecs, etc,
converting between Zarr V2 and V3, and
creating custom data types.

Examples can be run with cargo run --example <EXAMPLE_NAME>.

Some examples require non-default features, which can be enabled with --all-features or --features <FEATURES>.
Some examples support a -- --usage-log argument to print storage API calls during execution.

§Crate Features

§Default

filesystem: Re-export zarrs_filesystem as zarrs::filesystem
ndarray: ndarray utility functions for Array.
Codecs: blosc, crc32c, gzip, sharding, transpose, zstd.

§Non-Default

async: an experimental asynchronous API for stores, Array, and Group.
- The async API is runtime-agnostic. This has some limitations that are detailed in the Array docs.
- The async API is not as performant as the sync API.
dlpack: adds convenience methods for DLPack tensor interop to Array
Codecs: bitround, bz2, fletcher32, gdeflate, pcodec, zfp, zlib.

§`zarrs` Ecosystem

The Zarr specification is inherently unstable. It is under active development and new extensions are continually being introduced.

The zarrs crate has been split into multiple crates to:

allow external implementations of stores and extensions points to target a relatively stable API compatible with a range of zarrs versions,
enable automatic backporting of metadata compatibility fixes and changes due to standardisation,
stay up-to-date with unstable public dependencies (e.g. opendal, object_store, icechunk, etc) without impacting the release cycle of zarrs, and
improve compilation times.

A hierarchical overview of these crates can be found in the The zarrs Book.

§Core

zarrs: The core library for manipulating Zarr hierarchies.
zarrs_metadata: Zarr metadata support (re-exported as zarrs::metadata).
zarrs_metadata_ext: Zarr extensions metadata support (re-exported as zarrs::metadata_ext).
zarrs_data_type: The data type extension API for zarrs (re-exported in zarrs::array::data_type).
zarrs_storage: The storage API for zarrs (re-exported as zarrs::storage).
zarrs_plugin: The plugin API for zarrs (re-exported as zarrs::plugin).
zarrs_registry: The Zarr extension point registry for zarrs (re-exported as zarrs::registry).

§Stores

zarrs_filesystem: A filesystem store (re-exported as zarrs::filesystem).
zarrs_object_store: object_store store support.
zarrs_opendal: opendal store support.
zarrs_http: A synchronous http store.
zarrs_zip: A storage adapter for zip files.
zarrs_icechunk: icechunk store support.
- git-like version control for Zarr hierachies.
- Read “virtual Zarr datacubes” of archival formats (e.g., netCDF4, HDF5, etc.) created by VirtualiZarr and backed by icechunk.

§Bindings

zarrs-python: A high-performance codec pipeline for zarr-python.
zarrs_ffi: A subset of zarrs exposed as a C/C++ API.

§Zarr Metadata Conventions

ome_zarr_metadata: A library for OME-Zarr (previously OME-NGFF) metadata.

§Tools

zarrs_tools: Various tools for creating and manipulating Zarr V3 data with the zarrs rust crate
- A reencoder that can change codecs, chunk shape, convert Zarr V2 to V3, etc.
- Create an OME-Zarr hierarchy from a Zarr array.
- Transform arrays: crop, rescale, downsample, gradient magnitude, gaussian, noise filtering, etc.

§Benchmarks

zarr_benchmarks: Benchmarks of various Zarr V3 implementations: zarrs, zarr-python, tensorstore

§Licence

zarrs is licensed under either of

the Apache License, Version 2.0 LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0 or
the MIT license LICENSE-MIT or http://opensource.org/licenses/MIT, at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Re-exports§

pub use zarrs_metadata as metadata;
pub use zarrs_metadata_ext as metadata_ext;
pub use zarrs_plugin as plugin;
pub use zarrs_registry as registry;
pub use zarrs_storage as storage;
pub use zarrs_filesystem as filesystem;filesystem

Modules§

array: Zarr arrays.
array_subset: Array subsets.
byte_range: Byte ranges.
config: zarrs global configuration options.
group: Zarr groups.
node: Zarr nodes.
version: zarrs version information.

Crate zarrsCopy item path