Expand description
A catalog of every instrument data set on a file share, as open files the user owns.
openreadout index ROOT… -o INDEX_DIR walks the roots, reads only headers (never pixels,
samples or spectra), and writes three Parquet tables and a manifest into INDEX_DIR:
experiments.parquet: one row per data set (a file, a multi-file set such as a multi-file OME-TIFF or a CZI with its parts, or a directory data set such as a Bruker.d), with the experiment model flattened into columns (sample, instrument, technique term, method, operator, start time), image/table/trace/spectra summaries, integrity status and personal-data flags;files.parquet: one row per file seen, pointing at its data set (orunknown);problems.parquet: integrity findings, unreadable files, walk errors and personal-data flags;index.json:IndexManifest: roots, settings, crawl counters, totals per format, family, year and unknown extension, and the column list of every table.
The crawl is parallel (the rayon pool), resumable (a journal with checkpoints; a killed run
continues where it stopped), incremental (unchanged files are not reopened; moved files are
recognised by their content fingerprint), and its memory does not grow with the size of the
tree (records stream to disk; the walk holds one directory listing per level). See
crawl for the state files; multi-file grouping happens when the tables are written.
On top of an index: search() (a small query language, query), health()
(storage health report) and export_dataset (an ML-ready slice with a datasheet). pii finds personal
data in header metadata and redacts it. report builds the privacy-reviewed diagnostic
bundle of openreadout report for a file a reader refused or could not validate.
§Example
use openreadout_core::Registry;
use openreadout_index::{IndexOptions, index, search, SearchRequest};
let registry = Registry::new(); // the CLI registers every format reader here
let mut opts = IndexOptions::default();
opts.roots = vec!["/lab/share".into()];
opts.index_dir = "/lab/index".into();
let manifest = index(®istry, &opts, None)?;
println!("{} data sets", manifest.datasets);
let mut req = SearchRequest::default();
req.query = "format=czi objective=63x acquired<2020".into();
let hits = search("/lab/index".as_ref(), &req)?;
println!("{} matches", hits.total);Re-exports§
pub use crawl::CheckMode;pub use crawl::CrawlProgress;pub use crawl::IndexOptions;pub use crawl::index;pub use export::ExportDatasetOptions;pub use export::ExportDatasetReport;pub use export::export_dataset;pub use health::HealthOptions;pub use health::HealthReport;pub use health::health;pub use manifest::IndexManifest;pub use manifest::read_manifest;pub use search::SearchOutput;pub use search::SearchRequest;pub use search::search;
Modules§
- crawl
- The crawl: walk, read headers in parallel, journal, resume, reuse the previous run.
- export
export-dataset: one command turns a slice of an index into an ML-ready dataset.- fingerprint
- Content fingerprints (cheap, for moves and duplicate candidates) and full content hashes (for confirming duplicates).
- formats
- Format facts the index needs beyond the readers’ descriptors: how at risk a format is, which files count as open exports, and data-set stems for pairing an original with its export.
- group
- Multi-file data-set grouping, shared by the index and batch tables.
- health
- Storage health report over an index: integrity, unreadable formats, exact duplicates, near-duplicate exports, files at risk, totals and personal data.
- manifest
index.json: what an index directory holds, how it was made and what the crawl found. It is also the payload ofopenreadout index --jsonand of the MCP toolopenreadout_index.- pii
- Personal-data detection in header metadata, and redaction with stable salted hashes.
- query
- The search query language.
- record
- What the crawl records per walk item: one
Recordper file or directory data set (and per file no reader recognises). Records are journaled as JSON Lines while crawling and flattened into the Parquet tables (crate::tables) when the run is finalized. - report
openreadout report FILE: a privacy-reviewed diagnostic bundle for a file that a reader refused, failed on, or read without being able to validate it (assurance.levelother thanvalidated).- search
search: run a query (crate::query) overexperiments.parquet, streaming.- tables
- The three Parquet tables of an index, their columns, and generic reading and writing.
- walk
- A streaming, deterministic directory walk.
Functions§
- tables_
named - Columns of the table called
name(experiments,files,problems).