Skip to main content

Crate openreadout_index

Crate openreadout_index 

Source
Expand description

A catalog of every instrument data set on a file share, as open files the user owns.

openreadout index ROOT… -o INDEX_DIR walks the roots, reads only headers (never pixels, samples or spectra), and writes three Parquet tables and a manifest into INDEX_DIR:

  • experiments.parquet: one row per data set (a file, a multi-file set such as a multi-file OME-TIFF or a CZI with its parts, or a directory data set such as a Bruker .d), with the experiment model flattened into columns (sample, instrument, technique term, method, operator, start time), image/table/trace/spectra summaries, integrity status and personal-data flags;
  • files.parquet: one row per file seen, pointing at its data set (or unknown);
  • problems.parquet: integrity findings, unreadable files, walk errors and personal-data flags;
  • index.json: IndexManifest: roots, settings, crawl counters, totals per format, family, year and unknown extension, and the column list of every table.

The crawl is parallel (the rayon pool), resumable (a journal with checkpoints; a killed run continues where it stopped), incremental (unchanged files are not reopened; moved files are recognised by their content fingerprint), and its memory does not grow with the size of the tree (records stream to disk; the walk holds one directory listing per level). See crawl for the state files; multi-file grouping happens when the tables are written.

On top of an index: search() (a small query language, query), health() (storage health report) and export_dataset (an ML-ready slice with a datasheet). pii finds personal data in header metadata and redacts it. report builds the privacy-reviewed diagnostic bundle of openreadout report for a file a reader refused or could not validate.

§Example

use openreadout_core::Registry;
use openreadout_index::{IndexOptions, index, search, SearchRequest};

let registry = Registry::new(); // the CLI registers every format reader here
let mut opts = IndexOptions::default();
opts.roots = vec!["/lab/share".into()];
opts.index_dir = "/lab/index".into();
let manifest = index(&registry, &opts, None)?;
println!("{} data sets", manifest.datasets);
let mut req = SearchRequest::default();
req.query = "format=czi objective=63x acquired<2020".into();
let hits = search("/lab/index".as_ref(), &req)?;
println!("{} matches", hits.total);

Re-exports§

pub use crawl::CheckMode;
pub use crawl::CrawlProgress;
pub use crawl::IndexOptions;
pub use crawl::index;
pub use export::ExportDatasetOptions;
pub use export::ExportDatasetReport;
pub use export::export_dataset;
pub use health::HealthOptions;
pub use health::HealthReport;
pub use health::health;
pub use manifest::IndexManifest;
pub use manifest::read_manifest;
pub use search::SearchOutput;
pub use search::SearchRequest;
pub use search::search;

Modules§

crawl
The crawl: walk, read headers in parallel, journal, resume, reuse the previous run.
export
export-dataset: one command turns a slice of an index into an ML-ready dataset.
fingerprint
Content fingerprints (cheap, for moves and duplicate candidates) and full content hashes (for confirming duplicates).
formats
Format facts the index needs beyond the readers’ descriptors: how at risk a format is, which files count as open exports, and data-set stems for pairing an original with its export.
group
Multi-file data-set grouping, shared by the index and batch tables.
health
Storage health report over an index: integrity, unreadable formats, exact duplicates, near-duplicate exports, files at risk, totals and personal data.
manifest
index.json: what an index directory holds, how it was made and what the crawl found. It is also the payload of openreadout index --json and of the MCP tool openreadout_index.
pii
Personal-data detection in header metadata, and redaction with stable salted hashes.
query
The search query language.
record
What the crawl records per walk item: one Record per file or directory data set (and per file no reader recognises). Records are journaled as JSON Lines while crawling and flattened into the Parquet tables (crate::tables) when the run is finalized.
report
openreadout report FILE: a privacy-reviewed diagnostic bundle for a file that a reader refused, failed on, or read without being able to validate it (assurance.level other than validated).
search
search: run a query (crate::query) over experiments.parquet, streaming.
tables
The three Parquet tables of an index, their columns, and generic reading and writing.
walk
A streaming, deterministic directory walk.

Functions§

tables_named
Columns of the table called name (experiments, files, problems).