Crate zarrs_tools

source ·
Expand description

§zarrs_tools

Latest Version msrv build

Various tools for creating and manipulating Zarr v3 data with the zarrs rust crate.

zarrs_tools is experimental and in limited production use. Correctness issues with zarrs affecting past versions of zarrs_tools are detailed here.

Changelog (CHANGELOG.md)

§Tools

  • zarrs_reencode: reencode an array.
    • Can change the chunk size, shard size, codecs, fill value, chunk key encoding separator, and attributes.
  • zarrs_binary2zarr (feature binary2zarr): create an array from piped binary data.
  • zarrs_ncvar2zarr (feature ncvar2zarr): convert a netCDF variable to an array.
    • Supports multi-file datasets where a variable has been split along a single dimension.

The following tools are highly experimental and have had limited production testing:

  • zarrs_filter (feature filter): apply simple image filters (transformations) to an array.
  • zarrs_ome (feature ome): convert an array to OME-Zarr.
    • Outputs OME-Zarr 0.5-dev. This revision is currently recognised by Neuroglancer for Zarr V3.

See docs/ for tool documentation.

§zarrs Benchmarking

  • zarrs_reencode: suitable for round trip benchmarking.
  • zarrs_benchmark_read_sync (feature benchmark): benchmark the zarrs sync API.
  • zarrs_benchmark_read_async (feature benchmark): benchmark the zarrs async API.

See docs/benchmarks.md for some benchmark measurements.

§Install

§From crates.io

cargo install --all-features zarrs_tools

§From source

cargo install --all-features --path .
# cargo install --all-features --git https://github.com/LDeakin/zarrs_tools

§Enabling SIMD intrinsics

Encoding and decoding performance may be improved with avx2/sse2 enabled (if supported).

This can be enabled by compiling with either of:

  • RUSTFLAGS="-C target-cpu=native"
  • RUSTFLAGS="-C target-feature=+avx2,+sse2"

§Enabling non-default zarrs codecs

Non-default zarrs codecs (see zarrs crate features) can be enabled by passing them as feature flags. For example:

cargo install zarrs_tools --all-features --features zarrs/bitround,zarrs/zfp,zarrs/bz2,zarrs/pcodec

§Licence

zarrs_tools is licensed under either of

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.