Expand description
Parquet and Arrow IPC export of the non-image data of any OpenReadout dataset.
- Tables (FCS events, spike and event tables, plate reads, chromatography peak tables):
one column per table column in its stored type (
uint16,float32, …); platewellcolumns become a dictionary of well names. - Traces (electrophysiology sweeps, NMR FIDs and spectra, chromatograms): one row per
sample, a
sweepcolumn, the abscissa (time_s, or the spectrum’s axis such aschemical_shift_ppm) and onefloat64column per channel in physical units. - Spectra (mass spectrometry): long form, one row per point
(
scan, rt_s, ms_level, precursor_mz, mz, intensity), plus a per-scan summary written to a second file (<name>.scans.parquet).
Every column carries its unit (unit), label and the provenance of the normalized fields it
comes from in its field metadata; the file carries the source format, the instrument and a
JSON copy of info (keys in meta). The file is written under a temporary name, read back
through the Parquet or Arrow IPC reader and compared column by column (schema, metadata and
an xxh3 digest of every value), and only then renamed into place.
Compression is pure Rust: Snappy (Parquet default) or LZ4; Arrow IPC files are uncompressed unless LZ4 is asked for.
§Example
use std::path::Path;
use openreadout_arrow::{ColumnarFormat, ColumnarOptions, export_columnar};
use openreadout_core::{Dataset, Result};
/// Write table 0 (e.g. the events of an FCS file) as Parquet.
fn to_parquet(dataset: &mut dyn Dataset, input: &Path, output: &Path) -> Result<()> {
let mut options = ColumnarOptions::default();
options.format = ColumnarFormat::Parquet;
let report = export_columnar(dataset, input, output, &options)?;
println!("{} rows, {} columns, verified={}", report.rows_written, report.columns.len(), report.verified);
Ok(())
}Re-exports§
pub use arrow_array;pub use arrow_schema;
Modules§
- meta
- Field- and file-level metadata: units, labels, normalized-field provenance, the source format,
the instrument and a JSON copy of
info.
Structs§
- Columnar
Data - An export read into memory: the schema and its record batches (and, for spectra, the
per-scan summary). For in-process consumers such as the Python bindings (
File.to_arrow()). - Columnar
Export Report - Output of a Parquet or Arrow IPC export.
- Columnar
Options - What to write.
Enums§
- Columnar
Compression - Column compression. Every codec here is implemented in pure Rust.
- Columnar
Format - The container written.
- Columnar
Selection - Which part of the file to export.
Functions§
- default_
output - Default output path next to
input(orbase, its mirror under a batch output directory):<stem>.parquet,<stem>.table1.parquet,<stem>.trace2.sweep0.parquet,<stem>.run1.parquet, … (.arrowfor Arrow IPC). - export_
columnar - Export one table, trace or spectra run of
dstooutputas Parquet or Arrow IPC. Spectra also write the per-scan summary tosummary_output(output). Everything is written under temporary names, read back and verified, and only then renamed into place. - export_
size - How much an export with
optsreads, for progress reports: the rows of a table, the samples of a trace over the selected sweeps, or the spectra of a run. - ipc_
stream_ bytes - Serialize record batches as an Arrow IPC stream (for consumers that take bytes, e.g.
pyarrow.ipc.open_stream). - read_
columnar - Read one table, trace or spectra run into Arrow record batches, exactly as
export_columnarwould write them.max_rowscaps the rows read (an error when the selection is larger: passrows/sweepto narrow it). - read_
file - Read a Parquet or Arrow IPC file (e.g. one this crate wrote) into memory.
- summary_
output - The per-scan summary written beside a spectra export:
X.parquet→X.scans.parquet.