Skip to main content

Crate openreadout_arrow

Crate openreadout_arrow 

Source
Expand description

Parquet and Arrow IPC export of the non-image data of any OpenReadout dataset.

  • Tables (FCS events, spike and event tables, plate reads, chromatography peak tables): one column per table column in its stored type (uint16, float32, …); plate well columns become a dictionary of well names.
  • Traces (electrophysiology sweeps, NMR FIDs and spectra, chromatograms): one row per sample, a sweep column, the abscissa (time_s, or the spectrum’s axis such as chemical_shift_ppm) and one float64 column per channel in physical units.
  • Spectra (mass spectrometry): long form, one row per point (scan, rt_s, ms_level, precursor_mz, mz, intensity), plus a per-scan summary written to a second file (<name>.scans.parquet).

Every column carries its unit (unit), label and the provenance of the normalized fields it comes from in its field metadata; the file carries the source format, the instrument and a JSON copy of info (keys in meta). The file is written under a temporary name, read back through the Parquet or Arrow IPC reader and compared column by column (schema, metadata and an xxh3 digest of every value), and only then renamed into place.

Compression is pure Rust: Snappy (Parquet default) or LZ4; Arrow IPC files are uncompressed unless LZ4 is asked for.

§Example

use std::path::Path;

use openreadout_arrow::{ColumnarFormat, ColumnarOptions, export_columnar};
use openreadout_core::{Dataset, Result};

/// Write table 0 (e.g. the events of an FCS file) as Parquet.
fn to_parquet(dataset: &mut dyn Dataset, input: &Path, output: &Path) -> Result<()> {
    let mut options = ColumnarOptions::default();
    options.format = ColumnarFormat::Parquet;
    let report = export_columnar(dataset, input, output, &options)?;
    println!("{} rows, {} columns, verified={}", report.rows_written, report.columns.len(), report.verified);
    Ok(())
}

Re-exports§

pub use arrow_array;
pub use arrow_schema;

Modules§

meta
Field- and file-level metadata: units, labels, normalized-field provenance, the source format, the instrument and a JSON copy of info.

Structs§

ColumnarData
An export read into memory: the schema and its record batches (and, for spectra, the per-scan summary). For in-process consumers such as the Python bindings (File.to_arrow()).
ColumnarExportReport
Output of a Parquet or Arrow IPC export.
ColumnarOptions
What to write.

Enums§

ColumnarCompression
Column compression. Every codec here is implemented in pure Rust.
ColumnarFormat
The container written.
ColumnarSelection
Which part of the file to export.

Functions§

default_output
Default output path next to input (or base, its mirror under a batch output directory): <stem>.parquet, <stem>.table1.parquet, <stem>.trace2.sweep0.parquet, <stem>.run1.parquet, … (.arrow for Arrow IPC).
export_columnar
Export one table, trace or spectra run of ds to output as Parquet or Arrow IPC. Spectra also write the per-scan summary to summary_output(output). Everything is written under temporary names, read back and verified, and only then renamed into place.
export_size
How much an export with opts reads, for progress reports: the rows of a table, the samples of a trace over the selected sweeps, or the spectra of a run.
ipc_stream_bytes
Serialize record batches as an Arrow IPC stream (for consumers that take bytes, e.g. pyarrow.ipc.open_stream).
read_columnar
Read one table, trace or spectra run into Arrow record batches, exactly as export_columnar would write them. max_rows caps the rows read (an error when the selection is larger: pass rows/sweep to narrow it).
read_file
Read a Parquet or Arrow IPC file (e.g. one this crate wrote) into memory.
summary_output
The per-scan summary written beside a spectra export: X.parquet → X.scans.parquet.