Crate scientific_workflow 

Rust primitives for reproducible scientific workflows.

scientific-workflow provides the data and execution foundations needed to describe scientific systems, record their evolution, and organize scoped computational work. The crate is intentionally divided by responsibility: state representation, in-memory state time series, storage, orchestration, and language bridges remain separate modules rather than accumulating behind one monolithic interface.

Module boundaries (public API ownership)

The boundary map is strict: each module owns only one slice of behavior, and callers move data between boundaries without duplicating the same concern.

• study: declarative study/phase/task planning and run execution. It owns declaration validation, scheduling, cancellation, execution timing, and progress summaries. It does not own model semantics, storage formats, or schema declarations.
• configuration: strict study-level replicate settings, study-wide parameters.json, phase-scoped expansion, named paths, and resolved combinations. It owns input validation and parameter expansion only. It does not own task construction, state schemas, or persistence.
• system_state: typed heterogeneous fielded state values and schema.
• time_series: ordered in-memory complete-state collections for analysis.
• storage: asynchronous buffered persistence and completed-run reconstruction.
• execution: replicate subprocess dispatch, isolated output scopes, and directory-scoped recording path derivation.
• artifact: immutable input content-addressed publication under an execution scope, plus strict load-time verification.
• rng_record: lazy named replicate-seed derivation and validated reproducibility metadata for caller-owned RNG sources.
• prelude: curated import surfaces that preserve public boundaries.

Study vocabulary

A study::Study is the largest scope. It owns scheduling, cancellation, recording, and display for an ordered set of study::Phase values. A phase owns many study::Task values plus their concurrency, delay, timeout, dependency, and failure policies. A task owns one workload, which reports progress, detail, messages, and cancellation through study::TaskContext. Progress and one-shot work are modes of the same task type.

configuration is deliberately outside that hierarchy. It validates process-level replicate policy from study.json, resolves a study-wide parameters.json, then selects one string-keyed phase whose global, group-shared, and local choices expand into deterministic configuration::ResolvedConfiguration values. The downstream application decides how each combination becomes a task and owns all schemas, model inputs, storage, and other effects captured by the workload.

Supporting modules

execution dispatches isolated replicate subprocesses, creates their output scopes, and derives deterministic task recording paths. artifact atomically publishes and verifies content-addressed immutable bytes. rng_record stores validated RNG provenance while leaving random generation to applications.

system_state provides:

• JSON-defined, immutable field layouts;
• optional natural-language field descriptions without persisted Rust types;
• heterogeneous concrete Rust payloads behind a typed API;
• clone-free payload insertion, mutation, and extraction;
• explicit per-payload cloning of complete states;
• mutable, checked time-point progression.

Type erasure and boxing remain internal to that module. Consumer crates work with their original concrete payload types.

time_series provides the in-memory analysis collection for complete, ordered states. It enforces shared-layout identity and increasing simulation indices, offers a lightweight borrowed view, and permits field-level mutation without exposing mutable state time. It deliberately performs no serialization, chunking, or filesystem IO.

storage provides named partial-state streams with writer-owned sampling intervals, borrowed JSON encoding only when due, bounded asynchronous persistence through one worker per recording, byte-targeted chunking, atomic recording metadata, automatic operational timing, terminal summaries, name-selected payload decoders, and verified full-series or latest-state reconstruction. Import prelude::basics for these scientific primitives and prelude::study only at orchestration boundaries.

Basic use

use scientific_workflow::prelude::basics::*;

let spec = SystemStateSchema::load_json_template("state.json")?;
let mut state = spec.create_empty_state(SimulationTime::from_iteration(0));

assert!(
    state
        .insert_payload("population", vec![10_u64, 20, 30])?
        .is_none()
);
state
    .payload_mut::<Vec<u64>>("population")?
    .push(40);
let time = state.advance_simulation_time(None)?;
assert_eq!(time.iteration(), 1);
let population = state.take_payload::<Vec<u64>>("population")?;

assert_eq!(population, vec![10, 20, 30, 40]);

Future orchestration-layer features will organize scoped workflow execution without changing the public state-value ownership or storage contracts.

Release stability

This crate is a test release. Public API behavior is allowed to change across updates without backward compatibility guarantees.

Downstream no-overlap policy

For downstream consumers, preserve boundary ownership: keep orchestration in study, persistence in storage, and pure state in system_state/time_series. Do not implement overlapping behavior in a downstream layer; if a seam is missing, negotiate an explicit API addition.

Modules

artifact

Generic content-addressed input artifacts inside an execution scope.

configuration

Strict study settings, named paths, and phase-scoped parameter expansion.

execution

Process and filesystem execution boundaries for workflow runs.

prelude

Narrow end-user imports grouped by responsibility.

rng_record

Deterministic replicate seeds and provenance for application-owned RNGs.

storage

Recording persistence and reconstruction for scientific state samples.

study

Study, phase, and task orchestration.

system_state

Template-defined, heterogeneous scientific system states.

time_series

In-memory collections of ordered scientific system states.