oxide-batch

The public facade crate for OxideBatch, a restart-oriented batch processing framework for Rust inspired by Spring Batch.

This is the M5 Embedded Core Production Preview: a 0.x pre-1.0 release stabilizing the delivered M0-M4 embedded scope (durable PostgreSQL job/step/chunk execution, typed bounded fault tolerance, finite sequential/conditional flow, a guarded operator CLI, bounded local split/partitioning, and structured telemetry) rather than adding new batch capability. It is not 1.0, stable, GA, or enterprise-ready, and it does not claim full Spring Batch parity. See the Production Preview guide for what is and is not covered, and the M5 exit record for the release's evidence trail.

The facade owns validated job and step names, opaque instance/execution IDs, typed and value-redacted job parameters, canonical job-instance keys, lifecycle metadata, counters, exit statuses, redacted failure summaries, runtime-neutral repository ports, an in-memory reference repository, and async tasklet execution with cooperative stopping. Job and step listeners nest deterministically around tasklet work, and facade-owned lifecycle events expose only reviewed correlation and failure fields.

Job and step execution snapshots enforce the accepted lifecycle through database-agnostic optimistic versions. Exit status can be enriched independently of batch status, and restarting a stopped or failed record always creates a distinct STARTING attempt.

Repository operations are staged in an explicit unit of work and become visible only after commit. The in-memory implementation uses optimistic snapshot commits, reports a typed conflict when another unit commits first, and accepts injected clock and ID sources instead of reading hidden global state. It is intended for deterministic kernel tests and process-local execution; it is not durable across restarts.

With the optional postgres feature, PostgresJobRepository implements the same repository contract over the immutable OxideBatch schema. Runtime startup verifies metadata schema version 3 (direct upgrade from schemas 1 and 2; a schema-2 runtime rejects schema 3) but never applies migrations; deployments call PostgresMigrator separately with a migrator identity. See the upgrade and rollback guide before a schema upgrade. Production defaults require certificate and hostname validation, while plaintext transport is an explicit local-test choice. Connection strings, certificate paths, parameters, contexts, SQL text, and bound values are excluded from facade diagnostics.

[dependencies]
oxide-batch = { version = "0.5.0", features = ["postgres"] }

use std::sync::Arc;

use oxide_batch::{PostgresConfig, PostgresJobRepository, PostgresMigrator, SystemClock};

# async fn configure() -> Result<(), Box<dyn std::error::Error>> {
let migrator = PostgresConfig::new(std::env::var("MIGRATOR_DATABASE_URL")?)?;
PostgresMigrator::migrate(&migrator).await?;

let runtime = PostgresConfig::new(std::env::var("RUNTIME_DATABASE_URL")?)?;
let repository = PostgresJobRepository::connect(runtime, Arc::new(SystemClock)).await?;
# repository.close().await?;
# Ok(())
# }

The migrator and runtime connection strings should identify different least-privilege roles in production. The adapter remains behind the public repository and unit-of-work ports; SQLx types are never part of those APIs.

JobLauncher executes on an application-owned async runtime. The public Tasklet trait returns an OxideBatch-owned boxed future, so implementations can borrow call-scoped parameters and stop state without exposing Tokio types. Synchronous work must use BlockingTaskletAdapter, which applies an explicit nonzero concurrency bound and awaits already-running work before reporting a late stop.

Before-listeners run in registration order and after-listeners run in reverse order. A before failure prevents the associated user body. An after failure cannot undo completed user work; LaunchReport retains the provisional outcome and every redacted listener failure. Event sinks observe committed lifecycle states and cannot fail an execution, even if a sink panics.

M2 component contracts

The facade now owns runtime-neutral ItemReader, ItemProcessor, ItemWriter, and ChunkCompletion traits. Each asynchronous call may borrow component and call-scoped state without exposing an executor. Typed outcomes keep normal end-of-input, filtered items, cooperative stop, component failure, and post-commit acknowledgement separate.

Durable PostgreSQL writers receive a call-scoped BusinessTransaction port. Statements use separately bound, value-redacted facade types; SQLx pools, connections, transactions, rows, and errors remain adapter-internal.

Checkpoint and ExecutionContext are bounded, versioned, schema-aware JSON envelopes. Application codecs receive JSON object bytes rather than Serde types, own explicit old-version upgrade paths, and produce only redacted failure classifications. The default envelope limit is 64 KiB and depth 16, with hard ceilings aligned to the accepted metadata model.

M2 chunk execution

ChunkStep executes bounded read/process/filter/write attempts and publishes only counts returned by a successful ChunkTransaction commit. ChunkJob and JobLauncher::launch_chunk reuse the existing repository-backed job and step lifecycle. Stop before commit rolls the attempt back; stop or completion failure after commit cannot undo the committed chunk.

Chunk listeners nest in registration/reverse-registration order. Component and listener panic payloads are discarded, and an ambiguous transaction response becomes UNKNOWN rather than being inferred. Correlated chunk.started, chunk.committed, chunk.rolled_back, and chunk.unknown events include a nonzero chunk sequence.

With the postgres feature, PostgresChunkTransactionManager binds a launched chunk to its job/step execution identity. A PostgresChunkStateProvider produces the checkpoint and context for each commit from the prior durable counters and current checked counts. The adapter lends the writer one BusinessTransaction, then commits its writes, checkpoint, context, cumulative counters, injected-clock update time, and optimistic step version on the same connection. A failed CAS rolls everything back; a failed COMMIT acknowledgement discards the connection and reports UNKNOWN.

Managers that do not enlist the writer retain the documented at-least-once boundary and cannot claim PostgreSQL same-resource atomicity.

M2 durable restart and recovery

TaskletJob::new and ChunkJob::new require application-owned definition and component revisions. Chunk definitions also declare checkpoint/context schema IDs and versions plus their delivery mode. OxideBatch builds a bounded canonical manifest, hashes it with SHA-256, and binds every PostgreSQL execution to the exact persisted definition. Reusing one revision with different restart-relevant content is a typed definition-drift error. A failed or stopped execution can restart with the same digest or with one registered, directed DefinitionUpgrade; edges are not inferred or treated as transitive.

The current directed upgrade path copies checkpoint and context bytes unchanged. It is valid only when the mapped source and target steps retain the same state schemas and meanings. A state-schema change requires a future explicit transformation contract and is not silently accepted.

A PostgreSQL restart creates new job and step execution IDs and loads the latest committed checkpoint, execution context, and cumulative counters into the new step attempt. Completed and abandoned instances remain terminal. STARTING, STARTED, STOPPING, and UNKNOWN attempts block replay until an operator submits a versioned RecoveryRequest. Recovery rereads the durable row under lock, atomically appends prior/result status, reason, operator correlation, evidence digest, and injected-clock time, and changes the execution to FAILED or ABANDONED. Authentication, authorization, external evidence storage, and a recovery CLI remain deployment or later-operator concerns.

The PostgreSQL 15 and 18 CI axes terminate a separate worker process before and after the second chunk commit, inspect durable state through a new connection, audit the orphan, and finish through a distinct restart attempt. See the transaction guarantee and crash/recovery runbook for the exact replay boundary and operator procedure.

First in-memory job

The checked-in example defines a tasklet using only facade-owned batch types. The application supplies the async executor and explicitly constructs the process-local repository, clock, and identifier source:

cargo run -p oxide-batch --example first_job

It prints the tasklet observation and the final persisted job status. The in-memory repository is intentionally non-durable; use this M1 example for learning, local execution, and deterministic tests rather than restart guarantees across process failure.

PostgreSQL local partitions

The checked-in PostgreSQL example owns a tasklet-only partition definition and uses a finite one-worker/two-connection manifest budget. The URLs below are local-example credentials; production deployments use separate least-privilege migrator and runtime identities with verified TLS.

export OXIDEBATCH_EXAMPLE_MIGRATOR_URL='postgres://...'
export OXIDEBATCH_EXAMPLE_RUNTIME_URL='postgres://...'
cargo run -p oxide-batch --features postgres --example postgres_local_partition -- migrate
cargo run -p oxide-batch --features postgres --example postgres_local_partition -- launch
cargo run -p oxide-batch --features postgres --example postgres_local_partition -- inspect

The interruption path exits after alpha commits and beta starts. Inspect the durable rows, record an audited recovery, and restart; alpha is carried forward and is not invoked again.

cargo run -p oxide-batch --features postgres --example postgres_local_partition -- interrupt
cargo run -p oxide-batch --features postgres --example postgres_local_partition -- inspect
cargo run -p oxide-batch --features postgres --example postgres_local_partition -- recover
cargo run -p oxide-batch --features postgres --example postgres_local_partition -- restart
cargo run -p oxide-batch --features postgres --example postgres_local_partition -- inspect