Crate oximedia_batch 

Comprehensive batch processing engine for OxiMedia

This crate provides a production-ready batch processing system with:

• Job queuing and scheduling
• Worker pool management
• Template-based configuration
• Watch folder automation
• Distributed processing support
• REST API and CLI interfaces

Re-exports

pub use error::BatchError;

pub use error::Result;

pub use job::BatchJob;

pub use job::BatchOperation;

pub use job::InputSpec;

pub use job::OutputSpec;

pub use types::JobId;

pub use types::JobState;

pub use types::Priority;

pub use types::RetryPolicy;

Modules

audit_log

Immutable append-only audit trail for all job lifecycle events. Audit log: track who submitted, modified, and cancelled each job.

batch_analytics

Per-job and fleet-wide batch analytics: throughput, latency, error rates. Batch analytics — run-time trend analysis, failure rates, throughput metrics, and SLA monitoring for batch processing workloads.

batch_report

Batch job reporting and summary aggregation.

batch_runner

Batch job runner: configuration, item tracking, and run-level statistics.

batch_schedule

Batch scheduling — schedule types, individual schedules, and a scheduler registry.

chaining

Job chaining: define DAG-like sequential/parallel chains between jobs. Job chaining — dependency-aware sequential job dispatch.

checkpoint

Durable checkpoint persistence for mid-job progress and crash recovery. Periodic state checkpointing for crash recovery.

checkpointing

Job checkpointing: save and restore execution state for resume-on-failure.

cluster_discovery

Cluster-member discovery and heartbeat tracking for distributed workers. Cluster discovery for auto-detecting batch workers on the network.

conditional_dag

Conditional DAG with result-based branch routing.

cost_estimator

CPU/GPU/IO cost estimation for job admission control and resource planning. Cost estimator: predict job duration and resource usage from historical data.

dead_letter_queue

Dead-letter queue for failed jobs with replay and quarantine support. Dead letter queue for permanently failed jobs.

dep_graph

Simple dependency graph for batch jobs using numeric IDs.

dependency

Job dependency graph with topological sort and cycle detection.

error

Error types for batch processing

error_recovery

Error recovery strategies: retry with backoff, circuit breaker, fallback. Batch error recovery policy.

fair_scheduler

Fair scheduler: prevent starvation of low-priority jobs.

graceful_shutdown

Graceful shutdown coordinator: drain window, force-cancel, status reporting. Graceful shutdown for BatchEngine.

job

Job definition and configuration

job_archive

Archive completed batch jobs for historical querying and analytics.

job_deps

Job dependency tracker: declarative before/after constraint resolution. Job dependency chaining with topological sort and cycle detection.

job_migration

Job migration: upgrade job schemas when template format changes.

job_splitting

Job splitting: automatically partition large transcode jobs across workers.

metrics

Prometheus metrics integration

monitoring

Monitoring and progress tracking

notification_hub

Notification hub: fan-out job events to webhooks, email, and Slack. Batch notification hub — webhook callbacks, email notification specs, and notification deduplication.

notifications

Notification system for job events

operations

Batch operation implementations

output_collector

Output collection — gather, classify, and summarise task output entries.

pipeline_validator

Validate batch pipeline configurations before execution.

presets

Job presets and templates

priority_queue

Priority-aware job queue backed by a binary max-heap.

processor

Generic BatchJob<T>, BatchQueue, JobExecutor, TranscodeJobSpec, BatchProcessor, and JobStats.

progress_agg

Progress aggregator: roll up subtask progress into parent-job percentage. Batch-level progress aggregation.

progress_tracker

Progress tracking for batch jobs.

queue

Job queue implementation

quota

Quota definition types: resource ceilings and usage counters. Resource quota management for batch processing.

quota_enforcer

Quota enforcer: per-user/team hard limits on concurrent and total jobs. Quota enforcement — per-user/tenant job limits, storage quotas, CPU hour budgets, and violation handling.

rate_limiter

Rate limiting primitives for batch processing pipelines.

resource_estimator

Resource estimation for batch processing jobs.

resource_reservation

Resource reservation: pre-allocate GPU/CPU cores for high-priority jobs.

retry_policy

Retry policies and tracking for batch job execution.

task_group

Task group management — group policies, task collections, and aggregate results.

template

Template system for dynamic file naming and configuration

throttle

Rate throttling for batch job execution.

timeout_enforcer

Per-job timeout tracking and enforcement.

types

Core types for batch processing

utils

Utility functions and helpers for batch processing

work_stealing

Work-stealing scheduler for load balancing across workers.

Structs

ShutdownConfig

Configuration for graceful shutdown of [BatchEngine].

ShutdownFlag

Shared shutdown flag — stored as an AtomicU8 to allow lock-free reads.

ShutdownReport

Summary returned by [BatchEngine::request_shutdown].

Enums

ShutdownState

Shutdown progression state.