zarrs_tools  

Various tools for creating and manipulating Zarr v3 data with the zarrs rust crate.

zarrs is experimental and in limited production use. Correctness issues with zarrs affecting past versions of zarrs_tools are detailed here.

Changelog (CHANGELOG.md)

Tools

• Benchmark scripts that measure the time to retrieve an array either chunk-by-chunk or at once into a single array.
  • zarrs_benchmark_read_sync: benchmark with the zarrs sync API.
  • zarrs_benchmark_read_async: benchmark with the zarrs async API.
  • See docs/benchmarks.md for some benchmark measurements.
• zarrs_reencode: reencode a Zarr v3 array. Example.
  • Can change the chunk size, shard size, compression, etc.
  • Suitable for round trip benchmarking.
• zarrs_binary2zarr: create a Zarr v3 array from piped binary data. Example.
• zarrs_ncvar2zarr (requires ncvar2zarr feature): convert a netCDF variable to a zarr V3 array. Example.

Install

From crates.io

cargo install zarrs_tools --all-features

From source

cargo install --path .  --all-features

Enabling SIMD intrinsics

Encoding and decoding performance may be improved with avx2/sse2 enabled (if supported).

This can be enabled by compiling with either of:

• RUSTFLAGS="-C target-cpu=native"
• RUSTFLAGS="-C target-feature=+avx2,+sse2"

Enabling non-default zarrs codecs

Non-default zarrs codecs (see zarrs crate features) can be enabled by passing them as feature flags.

For example:

cargo install zarrs_tools --features zarrs/bitround,zarrs/zfp,zarrs/bz2,zarrs/pcodec

Licence

zarrs_tools is licensed under either of

• the Apache License, Version 2.0 LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0 or
• the MIT license LICENSE-MIT or http://opensource.org/licenses/MIT, at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.