Crate oxigdal_zarr

Expand description

OxiGDAL Zarr Driver - Pure Rust Zarr v2/v3 Support

This crate provides a pure Rust implementation of the Zarr storage specification for chunked, compressed, N-dimensional arrays. It supports both Zarr v2 and v3 specifications with various storage backends and compression codecs.

§Features

Zarr Versions: Full support for both Zarr v2 and v3 specifications
Storage Backends: Filesystem, S3, HTTP, and in-memory storage
Compression: Blosc, Zstd, Gzip, LZ4 codecs
Filters: Shuffle, Delta, Scale-offset filters
Async I/O: Async support for cloud storage backends
Parallel: Parallel chunk reading and writing
Caching: LRU caching for chunks

§Zarr Format

Zarr is a format for the storage of chunked, compressed, N-dimensional arrays. It was designed for use in parallel computing and is widely used in scientific computing, particularly for earth observation and climate data.

§Key Concepts

Array: N-dimensional data structure
Chunk: Fixed-size sub-array for storage
Codec: Compression/encoding method
Filter: Data transformation before/after codec
Group: Hierarchical organization of arrays
Attributes: Metadata attached to arrays/groups

§Example - Reading Zarr Array

use oxigdal_zarr::{ZarrReader, FilesystemStore};
use oxigdal_zarr::metadata::v2::ArrayMetadataV2;

// Open a Zarr v2 array
let store = FilesystemStore::open("data.zarr")?;
let reader = ZarrReader::open_v2(store)?;

println!("Shape: {:?}", reader.shape());
println!("Chunks: {:?}", reader.chunks());
println!("Data type: {:?}", reader.dtype());

// Read a chunk
let chunk_coords = vec![0, 0, 0];
let chunk_data = reader.read_chunk(&chunk_coords)?;

// Read a slice
let slice = reader.read_slice(&[0..10, 0..20, 0..30])?;

§Example - Writing Zarr Array

use oxigdal_zarr::{ZarrWriter, FilesystemStore};
use oxigdal_zarr::metadata::v2::{ArrayMetadataV2, DType};
use oxigdal_zarr::codecs::Compressor;

// Create a new Zarr v2 array
let store = FilesystemStore::create("output.zarr")?;
let metadata = ArrayMetadataV2 {
    shape: vec![100, 200, 300],
    chunks: vec![10, 20, 30],
    dtype: DType::Float32,
    compressor: Some(Compressor::Zstd { level: 3 }),
    fill_value: 0.0,
    order: 'C',
    filters: None,
};

let mut writer = ZarrWriter::create_v2(store, metadata)?;

// Write a chunk
let chunk_coords = vec![0, 0, 0];
let chunk_data = vec![0.0f32; 10 * 20 * 30];
writer.write_chunk(&chunk_coords, &chunk_data)?;

writer.finalize()?;

§Storage Backends

§Filesystem

use oxigdal_zarr::FilesystemStore;

let store = FilesystemStore::open("data.zarr")?;

§S3

use oxigdal_zarr::S3Store;

let store = S3Store::new("bucket-name", "prefix/data.zarr").await?;

§HTTP

use oxigdal_zarr::HttpStore;

let store = HttpStore::new("https://example.com/data.zarr")?;

Re-exports§

pub use chunk::ChunkCoord;
pub use chunk::ChunkGrid;
pub use chunk::ChunkIndex;
pub use consolidation::ConsolidatedMetadata;
pub use consolidation::ConsolidatedStore;
pub use consolidation::consolidate_metadata;
pub use dimension::Dimension;
pub use dimension::DimensionSeparator;
pub use dimension::Shape;
pub use error::Result;
pub use error::ZarrError;
pub use reader::v3::ZarrV3Reader;
pub use reader::ZarrReader;
pub use reader::ZarrReaderV2;
pub use storage::Store;
pub use storage::StoreKey;
pub use writer::v3::ZarrV3Writer;
pub use writer::ZarrWriter;
pub use writer::ZarrWriterV2;
pub use storage::filesystem::FilesystemStore;
pub use storage::memory::MemoryStore;

Modules§

chunk: Chunk coordinate and grid utilities for Zarr arrays
codecs: Compression codecs for Zarr arrays
consolidation: Consolidated metadata support for Zarr arrays
dimension: Dimension and shape utilities for Zarr arrays
error: Error types for Zarr operations
filters: Data filters for Zarr arrays
metadata: Zarr metadata structures for v2 and v3 specifications
reader: Zarr array readers
sharding: Zarr v3 Sharding Extension
storage: Storage backends for Zarr arrays
transformers: Storage Transformers for Zarr v3
writer: Zarr array writers

Constants§

NAME: Crate name
VERSION: Crate version
ZARR_VERSION_2: Zarr specification version 2
ZARR_VERSION_3: Zarr specification version 3