Skip to main content

Crate oxigeo_zarr

Crate oxigeo_zarr 

Source
Expand description

OxiGeo Zarr Driver - Pure Rust Zarr v2/v3 Support

This crate provides a pure Rust implementation of the Zarr storage specification for chunked, compressed, N-dimensional arrays. It supports both Zarr v2 and v3 specifications with various storage backends and compression codecs.

§Features

  • Zarr Versions: Zarr v2 and v3 arrays (chunk read/write, codec and filter pipelines, v3 sharding)
  • Storage Backends: Filesystem, S3, HTTP, and in-memory storage (with byte-range reads used for cloud-efficient shard access)
  • Compression: Zstd, Gzip, LZ4 codecs (all Pure Rust via OxiARC), plus the v3 crc32c checksum codec
  • Filters (opt-in cargo features): shuffle, delta, scale-offset
  • Async I/O: Async support for cloud storage backends
  • Parallel: Parallel chunk reading and writing
  • Caching: LRU caching for chunks

§Zarr Format

Zarr is a format for the storage of chunked, compressed, N-dimensional arrays. It was designed for use in parallel computing and is widely used in scientific computing, particularly for earth observation and climate data.

§Key Concepts

  • Array: N-dimensional data structure
  • Chunk: Fixed-size sub-array for storage
  • Codec: Compression/encoding method
  • Filter: Data transformation before/after codec
  • Group: Hierarchical organization of arrays
  • Attributes: Metadata attached to arrays/groups

§Example - Reading a Zarr v2 Array

use oxigeo_zarr::{ZarrReaderV2, FilesystemStore};

// Open a Zarr v2 array stored under the "temperature" path.
let store = FilesystemStore::open("data.zarr")?;
let reader = ZarrReaderV2::open(store, "temperature")?;

println!("Shape: {:?}", reader.shape());
println!("Chunks: {:?}", reader.chunks());
println!("Data type: {}", reader.dtype());

// Read one chunk (decoded element bytes in the array's stored order).
let chunk_data = reader.read_chunk(&[0, 0])?;

§Example - Writing a Zarr v2 Array

use oxigeo_zarr::{ZarrWriterV2, FilesystemStore};
use oxigeo_zarr::codecs::CompressorConfig;
use oxigeo_zarr::metadata::v2::ArrayMetadataV2;

let store = FilesystemStore::create("output.zarr")?;
let metadata = ArrayMetadataV2::new(vec![100, 200], vec![10, 20], "<f4")
    .with_compressor(CompressorConfig::Zstd { level: 3 });

let mut writer = ZarrWriterV2::create(store, "temperature", metadata)?;

// Write one 10x20 float32 chunk.
let chunk_data = vec![0u8; 10 * 20 * 4];
writer.write_chunk(&[0, 0], &chunk_data)?;
writer.finalize()?;

§Storage Backends

§Filesystem

ⓘ
use oxigeo_zarr::FilesystemStore;

let store = FilesystemStore::open("data.zarr")?;

§S3

ⓘ
use oxigeo_zarr::S3Store;

let store = S3Store::new("bucket-name", "prefix/data.zarr").await?;

§HTTP

ⓘ
use oxigeo_zarr::HttpStore;

let store = HttpStore::new("https://example.com/data.zarr")?;

Re-exports§

pub use chunk::ChunkCoord;
pub use chunk::ChunkGrid;
pub use chunk::ChunkIndex;
pub use consolidation::ConsolidatedMetadata;
pub use consolidation::ConsolidatedStore;
pub use consolidation::consolidate_metadata;
pub use dimension::Dimension;
pub use dimension::DimensionSeparator;
pub use dimension::Shape;
pub use error::Result;
pub use error::ZarrError;
pub use reader::ZarrReader;
pub use reader::ZarrReaderV2;
pub use reader::v3::ZarrV3Reader;
pub use storage::Store;
pub use storage::StoreKey;
pub use writer::ZarrWriter;
pub use writer::ZarrWriterV2;
pub use writer::v3::ZarrV3Writer;
pub use storage::filesystem::FilesystemStore;
pub use storage::memory::MemoryStore;
pub use storage::cache::CachingStorage;

Modules§

chunk
Chunk coordinate and grid utilities for Zarr arrays
codecs
Compression codecs for Zarr arrays
consolidation
Consolidated metadata support for Zarr arrays
dimension
Dimension and shape utilities for Zarr arrays
error
Error types for Zarr operations
filters
Data filters for Zarr arrays
metadata
Zarr metadata structures for v2 and v3 specifications
reader
Zarr array readers
sharding
Zarr v3 Sharding Extension
storage
Storage backends for Zarr arrays
transformers
Storage Transformers for Zarr v3
writer
Zarr array writers

Constants§

NAME
Crate name
VERSION
Crate version
ZARR_VERSION_2
Zarr specification version 2
ZARR_VERSION_3
Zarr specification version 3