Expand description
OxiGeo Zarr Driver - Pure Rust Zarr v2/v3 Support
This crate provides a pure Rust implementation of the Zarr storage specification for chunked, compressed, N-dimensional arrays. It supports both Zarr v2 and v3 specifications with various storage backends and compression codecs.
§Features
- Zarr Versions: Zarr v2 and v3 arrays (chunk read/write, codec and filter pipelines, v3 sharding)
- Storage Backends: Filesystem, S3, HTTP, and in-memory storage (with byte-range reads used for cloud-efficient shard access)
- Compression: Zstd, Gzip, LZ4 codecs (all Pure Rust via OxiARC),
plus the v3
crc32cchecksum codec - Filters (opt-in cargo features):
shuffle,delta,scale-offset - Async I/O: Async support for cloud storage backends
- Parallel: Parallel chunk reading and writing
- Caching: LRU caching for chunks
§Zarr Format
Zarr is a format for the storage of chunked, compressed, N-dimensional arrays. It was designed for use in parallel computing and is widely used in scientific computing, particularly for earth observation and climate data.
§Key Concepts
- Array: N-dimensional data structure
- Chunk: Fixed-size sub-array for storage
- Codec: Compression/encoding method
- Filter: Data transformation before/after codec
- Group: Hierarchical organization of arrays
- Attributes: Metadata attached to arrays/groups
§Example - Reading a Zarr v2 Array
use oxigeo_zarr::{ZarrReaderV2, FilesystemStore};
// Open a Zarr v2 array stored under the "temperature" path.
let store = FilesystemStore::open("data.zarr")?;
let reader = ZarrReaderV2::open(store, "temperature")?;
println!("Shape: {:?}", reader.shape());
println!("Chunks: {:?}", reader.chunks());
println!("Data type: {}", reader.dtype());
// Read one chunk (decoded element bytes in the array's stored order).
let chunk_data = reader.read_chunk(&[0, 0])?;§Example - Writing a Zarr v2 Array
use oxigeo_zarr::{ZarrWriterV2, FilesystemStore};
use oxigeo_zarr::codecs::CompressorConfig;
use oxigeo_zarr::metadata::v2::ArrayMetadataV2;
let store = FilesystemStore::create("output.zarr")?;
let metadata = ArrayMetadataV2::new(vec![100, 200], vec![10, 20], "<f4")
.with_compressor(CompressorConfig::Zstd { level: 3 });
let mut writer = ZarrWriterV2::create(store, "temperature", metadata)?;
// Write one 10x20 float32 chunk.
let chunk_data = vec![0u8; 10 * 20 * 4];
writer.write_chunk(&[0, 0], &chunk_data)?;
writer.finalize()?;§Storage Backends
§Filesystem
ⓘ
use oxigeo_zarr::FilesystemStore;
let store = FilesystemStore::open("data.zarr")?;§S3
ⓘ
use oxigeo_zarr::S3Store;
let store = S3Store::new("bucket-name", "prefix/data.zarr").await?;§HTTP
ⓘ
use oxigeo_zarr::HttpStore;
let store = HttpStore::new("https://example.com/data.zarr")?;Re-exports§
pub use chunk::ChunkCoord;pub use chunk::ChunkGrid;pub use chunk::ChunkIndex;pub use consolidation::ConsolidatedMetadata;pub use consolidation::ConsolidatedStore;pub use consolidation::consolidate_metadata;pub use dimension::Dimension;pub use dimension::DimensionSeparator;pub use dimension::Shape;pub use error::Result;pub use error::ZarrError;pub use reader::ZarrReader;pub use reader::ZarrReaderV2;pub use reader::v3::ZarrV3Reader;pub use storage::Store;pub use storage::StoreKey;pub use writer::ZarrWriter;pub use writer::ZarrWriterV2;pub use writer::v3::ZarrV3Writer;pub use storage::filesystem::FilesystemStore;pub use storage::memory::MemoryStore;pub use storage::cache::CachingStorage;
Modules§
- chunk
- Chunk coordinate and grid utilities for Zarr arrays
- codecs
- Compression codecs for Zarr arrays
- consolidation
- Consolidated metadata support for Zarr arrays
- dimension
- Dimension and shape utilities for Zarr arrays
- error
- Error types for Zarr operations
- filters
- Data filters for Zarr arrays
- metadata
- Zarr metadata structures for v2 and v3 specifications
- reader
- Zarr array readers
- sharding
- Zarr v3 Sharding Extension
- storage
- Storage backends for Zarr arrays
- transformers
- Storage Transformers for Zarr v3
- writer
- Zarr array writers
Constants§
- NAME
- Crate name
- VERSION
- Crate version
- ZARR_
VERSION_ 2 - Zarr specification version 2
- ZARR_
VERSION_ 3 - Zarr specification version 3