Skip to main content

Crate zarrs

Crate zarrs 

Source
Expand description

zarrs is Rust library for the Zarr storage format for multidimensional arrays and metadata.

If you are a Python user, check out zarrs-python. It includes a high-performance codec pipeline for the reference zarr-python implementation.

zarrs supports Zarr V3 and a V3 compatible subset of Zarr V2. It is fully up-to-date and conformant with the Zarr 3.1 specification with support for:

  • all core extensions (data types, codecs, chunk grids, chunk key encodings, storage transformers),
  • all accepted Zarr Enhancement Proposals (ZEPs) and several draft ZEPs:
    • ZEP 0003: Variable chunking
    • ZEP 0007: Strings
    • ZEP 0009: Zarr Extension Naming
  • various registered extensions from zarr-developers/zarr-extensions/,
  • experimental codecs intended for future registration, and
  • user-defined custom extensions and stores.

A changelog can be found here. Correctness issues with past versions are detailed here.

Developed at the Department of Materials Physics, Australian National University, Canberra, Australia.

§Getting Started

§Zarr Version Support

zarrs has first-class Zarr V3 support and additionally supports a compatible subset of Zarr V2 data that:

  • can be converted to V3 with only a metadata change, and
  • uses array metadata that is recognised and supported for encoding/decoding.

zarrs supports forward conversion from Zarr V2 to V3. See Converting Zarr V2 to V3 in The zarrs Book, or try the zarrs_reencode CLI tool.

§Array Extension Support

Extensions are grouped into three categories:

  • Core: defined in the Zarr V3 specification and are fully supported.
  • Registered: specified at https://github.com/zarr-developers/zarr-extensions/
    • Registered extensions listed in the below tables are fully supported unless otherwise indicated.
  • Experimental: indicated by 🚧 in the tables below and recommended for evaluation only.
    • Experimental extensions are either pending registration or have no formal specification outside of the zarrs docs.
    • Experimental extensions may be unrecognised or incompatible with other Zarr implementations.
    • Experimental extensions may change in future releases without maintaining backwards compatibility.
  • Deprecated: indicated by strikethrough in the tables below
    • Deprecated aliases will not be removed, but are not recommended for use in new arrays.
    • Deprecated extensions may be removed in future releases.

zarrs will persist extension names if opening an existing array of creating an array from metadata.

§Data Types
DataTypeV3 data_type nameV2 dtypeElementOwned / Element
(Feature Flag)
Boolbool|b1bool
Int2int2i8
Int4int4i8
Int8int8|i1i8
Int16int16>i2 <i2i16
Int32int32>i4 <i4i32
Int64int64>i8 <i8i64
UInt2uint2u8
UInt4uint4u8
UInt8uint8|u1u8
UInt16uint16>u2 <u2u16
UInt32uint32>u4 <u4u32
UInt64uint64>u8 <u8u64
Float4E2M1FN†float4_e2m1fnmicrofloat::f4e2m1fn (microfloat)
Float6E2M3FN†float6_e2m3fnmicrofloat::f6e2m3fn (microfloat)
Float6E3M2FN†float6_e3m2fnmicrofloat::f6e3m2fn (microfloat)
Float8E3M4†float8_e3m4microfloat::f8e3m4 (microfloat)
Float8E4M3†float8_e4m3microfloat::f8e4m3 (microfloat)
float8::F8E4M3 (float8)
Float8E4M3B11FNUZ†float8_e4m3b11fnuzmicrofloat::f8e4m3b11fnuz (microfloat)
Float8E4M3FNUZ†float8_e4m3fnuzmicrofloat::f8e4m3fnuz (microfloat)
Float8E5M2†float8_e5m2microfloat::f8e5m2 (microfloat)
float8::F8E5M2 (float8)
Float8E5M2FNUZ†float8_e5m2fnuzmicrofloat::f8e5m2fnuz (microfloat)
Float8E8M0FNU†float8_e8m0fnumicrofloat::f8e8m0fnu (microfloat)
BFloat16bfloat16half::bf16
Float16float16>f2 <f2half::f16
Float32float32>f4 <f4f32
Float64float64>f8 <f8f64
Complex64complex64>c8 <c8Complex<f32>
Complex128complex128>c16 <c16Complex<f64>
ComplexBFloat16complex_bfloat16Complex<half::bf16>
ComplexFloat16complex_float16Complex<half::f16>
ComplexFloat32complex_float32Complex<f32>
ComplexFloat64complex_float64Complex<f64>
ComplexFloat4E2M1FN†complex_float4_e2m1fnComplex<microfloat::f4e2m1fn> (microfloat)
ComplexFloat6E2M3FN†complex_float6_e2m3fnComplex<microfloat::f6e2m3fn> (microfloat)
ComplexFloat6E3M2FN†complex_float6_e3m2fnComplex<microfloat::f6e3m2fn> (microfloat)
ComplexFloat8E3M4†complex_float8_e3m4Complex<microfloat::f8e3m4> (microfloat)
ComplexFloat8E4M3†complex_float8_e4m3Complex<float8::F8E4M3> (float8)
Complex<microfloat::f8e4m3> (microfloat)
ComplexFloat8E4M3B11FNUZ†complex_float8_e4m3b11fnuzComplex<microfloat::f8e4m3b11fnuz> (microfloat)
ComplexFloat8E4M3FNUZ†complex_float8_e4m3fnuzComplex<microfloat::f8e4m3fnuz> (microfloat)
ComplexFloat8E5M2†complex_float8_e5m2Complex<float8::F8E5M2> (float8)
Complex<microfloat::f8e5m2> (microfloat)
ComplexFloat8E5M2FNUZ†complex_float8_e5m2fnuzComplex<microfloat::f8e5m2fnuz> (microfloat)
ComplexFloat8E8M0FNU†complex_float8_e8m0fnuComplex<microfloat::f8e8m0fnu> (microfloat)
Optional🚧zarrs.optionalOption
RawBitsr*|V*[u8; N] / &[u8; N]
FixedLengthUTF32fixed_length_utf32<U* >U*Vec<char> / &[char]
[char; N] / &[char; N]
Stringstring|OString / &str
Bytesbytes
binary
🚧variable_length_bytes		|VXVec<u8> / &[u8]
NumpyDateTime64numpy.datetime64i64
chrono::DateTime<Utc> (chrono)
jiff::Timestamp (jiff)
NumpyTimeDelta64numpy.timedelta64i64
chrono::TimeDelta (chrono)
jiff::SignedDuration (jiff)

† Additional features (e.g. microfloat / float8) may be required to parse floating point fill values. All subfloat types support hex string fill values.

§Codecs
Codec TypeV3 nameV2 idFeature Flag*
Array to Arraytranspose(implicit with "order": "F")transpose
🚧reshape-
🚧numcodecs.fixedscaleoffsetfixedscaleoffset
bitroundbitroundbitround
🚧zarrs.squeeze-
Array to Bytesbytes(implicit array-to-bytes)
🚧zarrs.optional-
sharding_indexed-sharding
🚧vlen-arrayvlen-array
vlen-bytesvlen-bytes
vlen-utf8vlen-utf8
packbits-
🚧numcodecs.pcodecpcodecpcodec
🚧numcodecs.zfpyzfpyzfp
🚧zarrs.vlen-
🚧zarrs.vlen_v2-
zfp-zfp
Bytes to Bytesbloscbloscblosc
crc32ccrc32ccrc32c
gzipgzipgzip
zstdzstdzstd
🚧numcodecs.adler32adler32adler32
🚧numcodecs.bz2bz2bz2
🚧numcodecs.fletcher32fletcher32fletcher32
🚧numcodecs.shuffleshuffle
🚧numcodecs.zlibzlibzlib
🚧zarrs.gdeflate-gdeflate

* Bolded feature flags are part of the default set of features.

zarrs supports arrays created with zarr-python 3.0.0+ and numcodecs 0.15.1+ with various numcodecs.zarr3 codecs.

§Chunk Grids
Chunk GridZEPV3V2Feature Flag
regularZEP0001✓✓
rectilinear✓
🚧rectangularZEP0003 (draft)✓
🚧zarrs.regular_bounded✓
§Chunk Key Encodings
Chunk Key EncodingZEPV3V2Feature Flag
defaultZEP0001✓
v2ZEP0001✓✓
🚧zarrs.default_suffix✓
§Storage Transformers

Zarr V3 does not currently define any storage transformers.

§Storage Support

zarrs supports a huge range of stores (including custom stores) via the zarrs_storage API.

† Re-exported in the zarrs::storage module.
‡ Re-exported as the zarrs::filesystem module.
* Support depends on the underlying store.

The opendal and object_store crates are popular Rust storage backends that are fully supported via zarrs_opendal and zarrs_object_store. These backends provide more feature complete HTTP stores than zarrs_http.

zarrs_icechunk implements the Icechunk transactional storage engine, a storage specification for Zarr that supports object_store stores.

The AsyncToSyncStorageAdapter enables some async stores to be used in a sync context.

§Logging

zarrs logs information and warnings using the log crate. A logging implementation must be enabled to capture logs. See the log crate documentation for more details.

§Examples

§Create and Read a Zarr Hierarchy


// Create a filesystem store
let store_path: PathBuf = "/path/to/hierarchy.zarr".into();
let store: zarrs::storage::ReadableWritableListableStorage = Arc::new(
    // zarrs::filesystem requires the filesystem feature
    zarrs::filesystem::FilesystemStore::new(&store_path)?
);

// Write the root group metadata
zarrs::group::GroupBuilder::new()
    .build(store.clone(), "/")?
    // .attributes(...)
    .store_metadata()?;

// Create a new sharded V3 array using the array builder
let array = zarrs::array::ArrayBuilder::new(
    vec![3, 4], // array shape
    vec![2, 2], // regular chunk (shard) shape
    zarrs::array::data_type::float32(),
    0.0f32, // fill value
)
.subchunk_shape(vec![2, 1]) // subchunk (inner chunk) shape
.bytes_to_bytes_codecs(vec![
    // GzipCodec requires the gzip feature
    Arc::new(zarrs::array::codec::GzipCodec::new(5)?),
])
.dimension_names(["y", "x"].into())
.attributes(serde_json::json!({"Zarr V3": "is great"}).as_object().unwrap().clone())
.build(store.clone(), "/array")?; // /path/to/hierarchy.zarr/array

// Store the array metadata
array.store_metadata()?;
println!("{}", serde_json::to_string_pretty(array.metadata())?);
// {
//     "zarr_format": 3,
//     "node_type": "array",
//     ...
// }

// Perform some write operations on the chunks
array.store_chunk(
    &[0, 1], // chunk index
    &[0.2f32, 0.3, 1.2, 1.3]
)?;
array.store_array_subset(
    &[1..3, 1..3], // array indices
    &ndarray::array![[-1.1f32, -1.2], [-2.1, -2.2]]
)?;
array.erase_chunk(&[1, 1])?;

// Retrieve all array elements as an ndarray
let array_all: ndarray::Array2<f32> = array.retrieve_array_subset(&[0..3, 0..4])?;
println!("{array_all:4}");
// [[ NaN,  NaN,  0.2,  0.3],
//  [ NaN, -1.1, -1.2,  1.3],
//  [ NaN, -2.1,  NaN,  NaN]]

// Retrieve a chunk directly
let array_chunk: ndarray::Array2<f32> = array.retrieve_chunk(
    &[0, 1], // chunk index
)?;
println!("{array_chunk:4}");
// [[  0.2,  0.3],
//  [ -1.2,  1.3]]

// Retrieve a subchunk
use zarrs::array::ArrayShardedReadableExt;
let shard_index_cache = zarrs::array::ArrayShardedReadableExtCache::new(&array);
let array_subchunk: ndarray::Array2<f32> = array.retrieve_subchunk_opt(
    &shard_index_cache,
    &[0, 3], // subchunk index
    &zarrs::array::CodecOptions::default(),
)?;
println!("{array_subchunk:4}");
// [[ 0.3],
//  [ 1.3]]

§Additional Examples

Various examples can be found in the examples/ directory of the zarrs repository that demonstrate:

  • creating and manipulating zarr hierarchies with various stores (sync and async), codecs, etc,
  • converting between Zarr V2 and V3, and
  • creating custom data types.

Examples can be run with cargo run --example <EXAMPLE_NAME>.

  • Some examples require non-default features, which can be enabled with --all-features or --features <FEATURES>.
  • Some examples support a -- --usage-log argument to print storage API calls during execution.

§Crate Features

§Default
§Non-Default
  • async: an experimental asynchronous API for stores, Array, and Group.
    • The async API is runtime-agnostic. This has some limitations that are detailed in the Array docs.
    • The async API is not as performant as the sync API.
  • Codecs: adler32, bitround, bz2, fletcher32, gdeflate, pcodec, zfp, zlib.
  • dlpack: adds convenience methods for DLPack tensor interop to Array.
  • Additional Element/ElementOwned implementations:
    • float8: add support for float8 subfloat data types.
    • microfloat: add support for microfloat subfloat data types.
    • jiff: add support for jiff time data types.
    • chrono: add support for chrono time data types.

§zarrs Ecosystem

The Zarr specification is inherently unstable. It is under active development and new extensions are continually being introduced.

The zarrs crate has been split into multiple crates to:

  • allow external implementations of stores and extensions points to target a relatively stable API compatible with a range of zarrs versions,
  • enable automatic backporting of metadata compatibility fixes and changes due to standardisation,
  • stay up-to-date with unstable public dependencies (e.g. opendal, object_store, icechunk, etc) without impacting the release cycle of zarrs, and
  • improve compilation times.

A hierarchical overview of these crates can be found in the The zarrs Book.

§Core
§Stores
§Bindings
§Zarr Metadata Conventions
§Tools
  • zarrs_tools: Various tools for creating and manipulating Zarr V3 data with the zarrs rust crate
    • A reencoder that can change codecs, chunk shape, convert Zarr V2 to V3, etc.
    • Create an OME-Zarr hierarchy from a Zarr array.
    • Transform arrays: crop, rescale, downsample, gradient magnitude, gaussian, noise filtering, etc.

§Benchmarks

  • zarr_benchmarks: Benchmarks of various Zarr V3 implementations: zarrs, zarr-python, tensorstore

§Licence

zarrs is licensed under either of

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Re-exports§

pub use zarrs_filesystem as filesystem;filesystem
pub use zarrs_metadata as metadata;
pub use zarrs_metadata_ext as metadata_ext;
pub use zarrs_plugin as plugin;
pub use zarrs_storage as storage;

Modules§

array
Zarr arrays.
config
zarrs global configuration options.
convert
Zarr V2 to V3 conversion.
group
Zarr groups.
hierarchy
Zarr hierarchies.
node
Zarr nodes.
version
zarrs version information.