Crate zarrs

Expand description

zarrs is Rust library for the Zarr storage format for multidimensional arrays and metadata.

If you are a Python user, check out zarrs-python. It includes a high-performance codec pipeline for the reference zarr-python implementation.

zarrs supports Zarr V3 and a V3 compatible subset of Zarr V2. It is fully up-to-date and conformant with the Zarr 3.1 specification with support for:

all core extensions (data types, codecs, chunk grids, chunk key encodings, storage transformers),
all accepted Zarr Enhancement Proposals (ZEPs) and several draft ZEPs:
- ZEP 0003: Variable chunking
- ZEP 0007: Strings
- ZEP 0009: Zarr Extension Naming
various registered extensions from zarr-developers/zarr-extensions/,
experimental codecs intended for future registration, and
user-defined custom extensions and stores.

A changelog can be found here. Correctness issues with past versions are detailed here.

Developed at the Department of Materials Physics, Australian National University, Canberra, Australia.

§Getting Started

Review the Zarr version support, array extension support (codecs, data types, etc.), storage support, and the zarrs ecosystem.
View the the examples below.
Read the documentation and The zarrs Book.

§Zarr Version Support

zarrs has first-class Zarr V3 support and additionally supports a compatible subset of Zarr V2 data that:

can be converted to V3 with only a metadata change, and
uses array metadata that is recognised and supported for encoding/decoding.

zarrs supports forward conversion from Zarr V2 to V3. See Converting Zarr V2 to V3 in The zarrs Book, or try the zarrs_reencode CLI tool.

§Array Extension Support

Extensions are grouped into three categories:

Core: defined in the Zarr V3 specification and are fully supported.
Registered: specified at https://github.com/zarr-developers/zarr-extensions/
- Registered extensions listed in the below tables are fully supported unless otherwise indicated.
Experimental: indicated by 🚧 in the tables below and recommended for evaluation only.
- Experimental extensions are either pending registration or have no formal specification outside of the zarrs docs.
- Experimental extensions may be unrecognised or incompatible with other Zarr implementations.
- Experimental extensions may change in future releases without maintaining backwards compatibility.
Deprecated: indicated by ~~strikethrough~~ in the tables below
- Deprecated aliases will not be removed, but are not recommended for use in new arrays.
- Deprecated extensions may be removed in future releases.

Extension names and aliases are configurable with Config::codec_aliases_v3_mut and similar methods for data types and Zarr V2. zarrs will persist extension names if opening an existing array of creating an array from metadata.

§Data Types

`DataType`	V3 `data_type` `name`	V2 `dtype`	`ElementOwned` / `Element` (Feature Flag)
`Bool`	`bool`	`\|b1`	`bool`
`Int2`	`int2`		`i8`
`Int4`	`int4`		`i8`
`Int8`	`int8`	`\|i1`	`i8`
`Int16`	`int16`	`>i2` `<i2`	`i16`
`Int32`	`int32`	`>i4` `<i4`	`i32`
`Int64`	`int64`	`>i8` `<i8`	`i64`
`UInt2`	`uint2`		`u8`
`UInt4`	`uint4`		`u8`
`UInt8`	`uint8`	`\|u1`	`u8`
`UInt16`	`uint16`	`>u2` `<u2`	`u16`
`UInt32`	`uint32`	`>u4` `<u4`	`u32`
`UInt64`	`uint64`	`>u8` `<u8`	`u64`
`Float4E2M1FN`†	`float4_e2m1fn`
`Float6E2M3FN`†	`float6_e2m3fn`
`Float6E3M2FN`†	`float6_e3m2fn`
`Float8E3M4`†	`float8_e3m4`
`Float8E4M3`†	`float8_e4m3`		`float8::F8E4M3` (`float8`)
`Float8E4M3B11FNUZ`†	`float8_e4m3b11fnuz`
`Float8E4M3FNUZ`†	`float8_e4m3fnuz`
`Float8E5M2`†	`float8_e5m2`		`float8::F8E5M2` (`float8`)
`Float8E5M2FNUZ`†	`float8_e5m2fnuz`
`Float8E8M0FNU`†	`float8_e8m0fnu`
`BFloat16`	`bfloat16`		`half::bf16`
`Float16`	`float16`	`>f2` `<f2`	`half::f16`
`Float32`	`float32`	`>f4` `<f4`	`f32`
`Float64`	`float64`	`>f8` `<f8`	`f64`
`Complex64`	`complex64`	`>c8` `<c8`	`Complex<f32>`
`Complex128`	`complex128`	`>c16` `<c16`	`Complex<f64>`
`ComplexBFloat16`	`complex_bfloat16`		`Complex<half::bf16>`
`ComplexFloat16`	`complex_float16`		`Complex<half::f16>`
`ComplexFloat32`	`complex_float32`		`Complex<f32>`
`ComplexFloat64`	`complex_float64`		`Complex<f64>`
`ComplexFloat4E2M1FN`†	`complex_float4_e2m1fn`
`ComplexFloat6E2M3FN`†	`complex_float6_e2m3fn`
`ComplexFloat6E3M2FN`†	`complex_float6_e3m2fn`
`ComplexFloat8E3M4`†	`complex_float8_e3m4`
`ComplexFloat8E4M3`†	`complex_float8_e4m3`		`Complex<float8::F8E4M3>` (`float8`)
`ComplexFloat8E4M3B11FNUZ`†	`complex_float8_e4m3b11fnuz`
`ComplexFloat8E4M3FNUZ`†	`complex_float8_e4m3fnuz`
`ComplexFloat8E5M2`†	`complex_float8_e5m2`		`Complex<float8::F8E5M2>` (`float8`)
`ComplexFloat8E5M2FNUZ`†	`complex_float8_e5m2fnuz`
`ComplexFloat8E8M0FNU`†	`complex_float8_e8m0fnu`
`RawBits`	`r*`		`[u8; N]` / `&[u8; N]`
`String`	`string`	`\|O`	`String` / `&str`
`Bytes`	`bytes` ~~`binary`~~ 🚧`variable_length_bytes`	`\|VX`	`Vec<u8>` / `&[u8]`
`NumpyDateTime64`	`numpy.datetime64`		`i64` `chrono::DateTime<Utc>` (`chrono`) `jiff::Timestamp` (`jiff`)
`NumpyTimeDelta64`	`numpy.timedelta64`		`i64` `chrono::TimeDelta` (`chrono`) `jiff::SignedDuration` (`jiff`)

^{† Additional features (e.g. float8) may be required to parse floating point fill values. All subfloat types support hex string fill values.}

§Codecs

Codec Type	V3 `name`	V2 `id`	Feature Flag*
Array to Array	`transpose`	`transpose`	transpose
	🚧`reshape`	-
	🚧`numcodecs.fixedscaleoffset`	`fixedscaleoffset`
	`bitround`	`bitround`	bitround
	🚧`zarrs.squeeze`	-
Array to Bytes	`bytes`	-
	`sharding_indexed`	-	sharding
	🚧`vlen-array`	`vlen-array`
	`vlen-bytes`	`vlen-bytes`
	`vlen-utf8`	`vlen-utf8`
	`packbits`	`packbits`
	🚧`numcodecs.pcodec`	`pcodec`	pcodec
	🚧`numcodecs.zfpy`	`zfpy`	zfp
	🚧`zarrs.vlen`	-
	🚧`zarrs.vlen_v2`	-
	`zfp`	-	zfp
Bytes to Bytes	`blosc`	`blosc`	blosc
	`crc32c`	`crc32c`	crc32c
	`gzip`	`gzip`	gzip
	`zstd`	`zstd`	zstd
	🚧`numcodecs.adler32`	`adler32`	adler32
	🚧`numcodecs.bz2`	`bz2`	bz2
	🚧`numcodecs.fletcher32`	`fletcher32`	fletcher32
	🚧`numcodecs.shuffle`	`shuffle`
	🚧`numcodecs.zlib`	`zlib`	zlib
	🚧`zarrs.gdeflate`	-	gdeflate

^{* Bolded feature flags are part of the default set of features.}

zarrs supports arrays created with zarr-python 3.0.0+ and numcodecs 0.15.1+ with various numcodecs.zarr3 codecs.

§Chunk Grids

Chunk Grid	ZEP	V3	V2
`regular`	ZEP0001	✓	✓
🚧`rectangular`	ZEP0003 (draft)	✓
🚧`zarrs.regular_bounded`		✓

§Chunk Key Encodings

Chunk Key Encoding	ZEP	V3	V2
`default`	ZEP0001	✓
`v2`	ZEP0001	✓	✓
🚧`zarrs.default_suffix`		✓

§Storage Transformers

Zarr V3 does not currently define any storage transformers.

§Storage Support

zarrs supports a huge range of stores (including custom stores) via the zarrs_storage API.

Store/Storage Adapter	ZEP	Read	Write	List	Sync	Async	Crate
MemoryStore		✓	✓	✓	✓		zarrs_storage^†
FilesystemStore	0001	✓	✓	✓	✓		zarrs_filesystem^‡
AsyncOpendalStore		✓*	✓*	✓*		✓	zarrs_opendal
AsyncObjectStore		✓*	✓*	✓*		✓	zarrs_object_store
AsyncIcechunkStore		✓*	✓*	✓*		✓	zarrs_icechunk
HTTPStore		✓			✓		zarrs_http
AsyncToSyncStorageAdapter		✓	✓	✓	✓		zarrs_storage^†
SyncToAsyncStorageAdapter		✓	✓	✓		✓	zarrs_storage^†
UsageLogStorageAdapter		✓	✓	✓	✓	✓	zarrs_storage^†
PerformanceMetricsStorageAdapter		✓	✓	✓	✓	✓	zarrs_storage^†
ZipStorageAdapter		✓		✓	✓		zarrs_zip

^{† Re-exported in the zarrs::storage module.}
^{‡ Re-exported as the zarrs::filesystem module.}
^{* Support depends on the underlying store.}

The opendal and object_store crates are popular Rust storage backends that are fully supported via zarrs_opendal and zarrs_object_store. These backends provide more feature complete HTTP stores than zarrs_http.

zarrs_icechunk implements the Icechunk transactional storage engine, a storage specification for Zarr that supports object_store stores.

The AsyncToSyncStorageAdapter enables some async stores to be used in a sync context.

§Logging

zarrs logs information and warnings using the log crate. A logging implementation must be enabled to capture logs. See the log crate documentation for more details.

§Examples

§Create and Read a Zarr Hierarchy


// Create a filesystem store
let store_path: PathBuf = "/path/to/hierarchy.zarr".into();
let store: zarrs::storage::ReadableWritableListableStorage = Arc::new(
    // zarrs::filesystem requires the filesystem feature
    zarrs::filesystem::FilesystemStore::new(&store_path)?
);

// Write the root group metadata
zarrs::group::GroupBuilder::new()
    .build(store.clone(), "/")?
    // .attributes(...)
    .store_metadata()?;

// Create a new sharded V3 array using the array builder
let array = zarrs::array::ArrayBuilder::new(
    vec![3, 4], // array shape
    vec![2, 2], // regular chunk (shard) shape
    zarrs::array::DataType::Float32,
    0.0f32, // fill value
)
.array_to_bytes_codec(Arc::new(
    // The sharding codec requires the sharding feature
    zarrs::array::codec::ShardingCodecBuilder::new(
        [2, 1].try_into()? // inner chunk shape
    )
    .bytes_to_bytes_codecs(vec![
        // GzipCodec requires the gzip feature
        Arc::new(zarrs::array::codec::GzipCodec::new(5)?),
    ])
    .build()
))
.dimension_names(["y", "x"].into())
.attributes(serde_json::json!({"Zarr V3": "is great"}).as_object().unwrap().clone())
.build(store.clone(), "/array")?; // /path/to/hierarchy.zarr/array

// Store the array metadata
array.store_metadata()?;
println!("{}", serde_json::to_string_pretty(array.metadata())?);
// {
//     "zarr_format": 3,
//     "node_type": "array",
//     ...
// }

// Perform some write operations on the chunks
array.store_chunk_elements::<f32>(
    &[0, 1], // chunk index
    &[0.2, 0.3, 1.2, 1.3]
)?;
array.store_array_subset_ndarray::<f32, _>(
    &[1, 1], // array index (start of subset)
    ndarray::array![[-1.1, -1.2], [-2.1, -2.2]]
)?;
array.erase_chunk(&[1, 1])?;

// Retrieve all array elements as an ndarray
let array_all = array.retrieve_array_subset_ndarray::<f32>(&array.subset_all())?;
println!("{array_all:4}");
// [[ NaN,  NaN,  0.2,  0.3],
//  [ NaN, -1.1, -1.2,  1.3],
//  [ NaN, -2.1,  NaN,  NaN]]

// Retrieve a chunk directly
let array_chunk = array.retrieve_chunk_ndarray::<f32>(
    &[0, 1], // chunk index
)?;
println!("{array_chunk:4}");
// [[  0.2,  0.3],
//  [ -1.2,  1.3]]

// Retrieve an inner chunk
use zarrs::array::ArrayShardedReadableExt;
let shard_index_cache = zarrs::array::ArrayShardedReadableExtCache::new(&array);
let array_inner_chunk = array.retrieve_inner_chunk_ndarray_opt::<f32>(
    &shard_index_cache,
    &[0, 3], // inner chunk index
    &zarrs::array::codec::CodecOptions::default(),
)?;
println!("{array_inner_chunk:4}");
// [[ 0.3],
//  [ 1.3]]

§Additional Examples

Various examples can be found in the examples/ directory of the zarrs repository that demonstrate:

creating and manipulating zarr hierarchies with various stores (sync and async), codecs, etc,
converting between Zarr V2 and V3, and
creating custom data types.

Examples can be run with cargo run --example <EXAMPLE_NAME>.

Some examples require non-default features, which can be enabled with --all-features or --features <FEATURES>.
Some examples support a -- --usage-log argument to print storage API calls during execution.

§Crate Features

§Default

filesystem: Re-export zarrs_filesystem as zarrs::filesystem.
ndarray: ndarray utility functions for Array.
Codecs: blosc, crc32c, gzip, sharding, transpose, zstd.

§Non-Default

async: an experimental asynchronous API for stores, Array, and Group.
- The async API is runtime-agnostic. This has some limitations that are detailed in the Array docs.
- The async API is not as performant as the sync API.
Codecs: adler32, bitround, bz2, fletcher32, gdeflate, pcodec, zfp, zlib.
dlpack: adds convenience methods for DLPack tensor interop to Array.
Additional Element/ElementOwned implementations:
- float8: add support for float8 subfloat data types.
- jiff: add support for jiff time data types.
- chrono: add support for chrono time data types.

§`zarrs` Ecosystem

The Zarr specification is inherently unstable. It is under active development and new extensions are continually being introduced.

The zarrs crate has been split into multiple crates to:

allow external implementations of stores and extensions points to target a relatively stable API compatible with a range of zarrs versions,
enable automatic backporting of metadata compatibility fixes and changes due to standardisation,
stay up-to-date with unstable public dependencies (e.g. opendal, object_store, icechunk, etc) without impacting the release cycle of zarrs, and
improve compilation times.

A hierarchical overview of these crates can be found in the The zarrs Book.

§Core

zarrs: The core library for manipulating Zarr hierarchies.
zarrs_metadata: Zarr metadata support (re-exported as zarrs::metadata).
zarrs_metadata_ext: Zarr extensions metadata support (re-exported as zarrs::metadata_ext).
zarrs_data_type: The data type extension API for zarrs (re-exported in zarrs::array::data_type).
zarrs_storage: The storage API for zarrs (re-exported as zarrs::storage).
zarrs_plugin: The plugin API for zarrs (re-exported as zarrs::plugin).
zarrs_registry: The Zarr extension point registry for zarrs (re-exported as zarrs::registry).

§Stores

zarrs_filesystem: A filesystem store (re-exported as zarrs::filesystem).
zarrs_object_store: object_store store support.
zarrs_opendal: opendal store support.
zarrs_http: A synchronous http store.
zarrs_zip: A storage adapter for zip files.
zarrs_icechunk: icechunk store support.
- git-like version control for Zarr hierachies.
- Read “virtual Zarr datacubes” of archival formats (e.g., netCDF4, HDF5, etc.) created by VirtualiZarr and backed by icechunk.

§Bindings

zarrs-python: A high-performance codec pipeline for zarr-python.
zarrs_ffi: A subset of zarrs exposed as a C/C++ API.

§Zarr Metadata Conventions

ome_zarr_metadata: A library for OME-Zarr (previously OME-NGFF) metadata.

§Tools

zarrs_tools: Various tools for creating and manipulating Zarr V3 data with the zarrs rust crate
- A reencoder that can change codecs, chunk shape, convert Zarr V2 to V3, etc.
- Create an OME-Zarr hierarchy from a Zarr array.
- Transform arrays: crop, rescale, downsample, gradient magnitude, gaussian, noise filtering, etc.

§Benchmarks

zarr_benchmarks: Benchmarks of various Zarr V3 implementations: zarrs, zarr-python, tensorstore

§Licence

zarrs is licensed under either of

the Apache License, Version 2.0 LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0 or
the MIT license LICENSE-MIT or http://opensource.org/licenses/MIT, at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Re-exports§

pub use zarrs_metadata as metadata;
pub use zarrs_metadata_ext as metadata_ext;
pub use zarrs_plugin as plugin;
pub use zarrs_registry as registry;
pub use zarrs_storage as storage;
pub use zarrs_filesystem as filesystem;filesystem

Modules§

array: Zarr arrays.
array_subset: Array subsets.
config: zarrs global configuration options.
group: Zarr groups.
hierarchy: Zarr hierarchies.
indexer: Generic indexer support.
node: Zarr nodes.
version: zarrs version information.

Crate zarrs

Crate zarrs Copy item path

§Getting Started

§Zarr Version Support

§Array Extension Support

§Data Types

§Codecs

§Chunk Grids

§Chunk Key Encodings

§Storage Transformers

§Storage Support

§Logging

§Examples

§Create and Read a Zarr Hierarchy

§Additional Examples

§Crate Features

§Default

§Non-Default

§zarrs Ecosystem

§Core

§Stores

§Bindings

§Zarr Metadata Conventions

§Tools

§Benchmarks

§Licence

Re-exports§

Modules§

Crate zarrs

§`zarrs` Ecosystem