Crate zarrs

A rust library for the Zarr V3 storage format for multidimensional arrays and metadata.

Developed at the Department of Materials Physics, Australian National University, Canberra, Australia.

zarrs is experimental and in limited production use. Use at your own risk!

A changelog can be found here.

A subset of zarrs is exposed as a C API in the zarrs-ffi crate.

Stability

Zarrs is unstable and will remain unstable in the near future as the API is refined and the Zarr V3 specification is finalised.

Implementation Status

• ZEP0001 - Zarr specification version 3
• ZEP0002 - Sharding codec (under review)
• ZEP0003 - Variable chunking (draft)
• Stores
  • Sync: filesystem, memory, http, zip
  • Async: filesystem, memory, HTTP, Google Cloud Storage, Amazon s3, Microsoft Azure Storage
• Data types: core data types, raw bits, float16, bfloat16 (spec issue)
• Chunk grids: regular, rectangular (draft)
• Chunk key encoding: default, v2
• Codecs:
  • array->array: transpose, bitround (experimental)
  • array->bytes: bytes (spec issue), sharding, zfp (experimental)
  • bytes->bytes: blosc, gzip, zstd (spec issue), crc32c checksum
• Storage transformers: usage_log, performance_metrics

Crate Features

The following crate features are enabled by default:

• Codecs: blosc, gzip, transpose, zstd, sharding, crc32c.
• Stores: http, zip.
• ndarray: adds ndarray utility functions to Array.

The following features are disabled by default:

• Codecs: bitround, zfp
• Stores: s3 (Amazon S3) gcp (Google Cloud), azure (Microsoft Azure)
• async (experimental): enable asynchronous stores and associated storage, array, and group methods.

Examples

Examples can be run with cargo run --example EXAMPLE_NAME

Sync API

• array_read_write: create an array, write its metadata, write chunks in parallel, delete a chunk, read the whole array, read a chunk, and partially read a subset.
• sharded_array_read_write: write and read a sharded array.
• rectangular_array_read_write: write and read an array with a rectangular chunk grid.
• zip_array_read_write: write an array to a filesystem, zip it, then read it from the zipped file.
• http_array_read: read an array over HTTP.

Async API

• async_array_read_write.
• async_http_array_read.

Licence

zarrs is licensed under either of

• the Apache License, Version 2.0 LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0 or
• the MIT license LICENSE-MIT or http://opensource.org/licenses/MIT, at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Modules

• array
  Zarr arrays.
• array_subset
  Array subsets.
• byte_range
  Byte ranges.
• group
  Zarr groups.
• metadata
  Utilities to support metadata with a name and optional configuration.
• node
  Zarr nodes.
• plugin
  Plugin utilities for supporting Zarr extension points.
• storage
  Zarr storage (stores and storage transformers).
• version
  Zarrs version information.

Macros

• object_store_implasync
  Implement the storage traits for an object store