Crate array_format 

array-format

A block-backed, footer-indexed container for storing many n-dimensional arrays in a single file.

The format uses a delta/overlay architecture: each flush produces a self-describing sidecar file that stacks on top of the base, recording only the chunks that changed. Reads fall through to older layers for unchanged chunks, and layers can be merged back into a single file with compact.

Features

• Store many arrays in one object (or a small set of related sidecar files).
• Append arrays and update individual chunks without rewriting the whole file.
• Per-block compression (LZ4, Zstd, or none) recorded in the block table, so readers need no configuration to decode a file.
• Chunked or single-chunk layouts with coordinate-addressed reads.
• Logical deletes with periodic compaction to reclaim space.
• Works with any object_store-compatible backend (local filesystem, S3, GCS, Azure).

Quick start

use array_format::{ArrayFile, FileConfig, Lz4Codec};
use ndarray::Array;

// An in-memory file; use `ArrayFile::create(store, path, config)` for on-disk.
let mut file = ArrayFile::create_memory(FileConfig::new(Lz4Codec)).await?;

// Define and write a 1-D f32 array.
file.define_array::<f32>("signal", vec!["t".into()], vec![4], None, None)?;
let data = Array::from_vec(vec![1.0f32, 2.0, 3.0, 4.0]).into_dyn();
file.write_array("signal", vec![0], data.view()).await?;

// Read it back — `vec![], vec![]` means "the whole array".
let out = file.read_array::<f32>("signal", vec![], vec![]).await?;
assert_eq!(out.len(), 4);

Architecture

The crate is organized in layers:

Layer	Purpose	Key types
0 — Core	Primitives	DType, ChunkAddress, BlockId, Error
1 — Metadata	Array description	MergedArrayMeta, FillValue
2 — Codecs	Compression extension point	CompressionCodec
3 — Runtime	Read / write / compact	ArrayFile

CompressionCodec is the extension point: implement it to plug in custom compression algorithms. Storage is provided through any object_store-compatible backend (passed to ArrayFile::create); for tests and ephemeral use, ArrayFile::create_memory uses object_store’s in-memory backend.

Re-exports

pub use array::ArrayElement;

pub use codec::CompressionCodec;

pub use codec::Lz4Codec;

pub use codec::NoCompression;

pub use codec::ZstdCodec;

pub use dtype::DType;

pub use error::Error;

pub use error::Result;

pub use file::ArrayFile;

pub use file::DEFAULT_BLOCK_TARGET_SIZE;

pub use file::DEFAULT_CACHE_CAPACITY;

pub use file::DEFAULT_IO_CACHE_CAPACITY;

pub use file::FileConfig;

pub use file::MergedArrayMeta;

pub use layout::AttributeValue;

pub use layout::FillValue;

pub use stats::ArrayStats;

pub use stats::StatValue;

pub use stats::StatsFile;

pub use timestamp::TimestampNs;

Modules

address

Block identifiers and chunk address types.

array

The ArrayElement trait: the element types that can be stored.

block

Block metadata stored in the footer.

codec

Compression codec trait and built-in implementations.

dtype

Data type definitions for array elements.

error

Error types for the array-format crate.

file

The runtime: ArrayFile, the top-level read/write/compact handle.

layout

Array layout definitions and array metadata.

stats

Per-array statistics stored in an optional .stats sidecar.

timestamp

Nanosecond-precision timestamp wrapper.

Structs

DeltaCache

Two-level cache shared across all delta layers in an ArrayFile.