Struct DdpRunConfig 

pub struct DdpRunConfig {

    pub elche: ElCheConfig,
    pub checkpoint_every: Option<usize>,
    pub snapshot_timeout_secs: u64,
    pub progressive_dispatch: Option<bool>,
    pub max_grad_norm: Option<f64>,
    pub vram_pool: bool,
    pub augment: usize,
    pub epoch_splits: usize,
    pub transform: Option<TransformFn>,
    pub vram_max_usage: f64,
    pub ram_max_usage: f64,
    pub gpu_ram_share: Option<f64>,
    pub sample_cache: bool,
    pub disk_stage_gb: u64,
    pub disk_stage_dir: Option<PathBuf>,
    pub timeline: Option<Arc<Timeline>>,
    pub lr_scale_ratio: f64,
    pub save_path: Option<String>,
    pub max_failure: Option<MaxFailureThreshold>,
    pub heartbeat_timeout_secs: Option<u64>,
    pub epoch_callback_policy: EpochCallbackPolicy,
    pub eval_every_epochs: Option<usize>,
    pub reports_per_epoch: Option<usize>,
    pub record_log_dir: Option<String>,
    pub max_log_size: Option<u64>,
    pub dashboard_html: Option<String>,
    pub dashboard_theme: Option<String>,
    pub scalar_reductions: Reductions,
    pub resume_from: Option<String>,
    pub checkpoint_at_epoch: Option<usize>,
}

Configuration for framework-managed DDP training.

All fields have sensible defaults. Use the builder methods to customize.

Fields

elche: ElCheConfig

The DDP coordination/convergence STRATEGY — the single source of truth for mode (canonical), cadence tuning (anchor / max_anchor / min_anchor / overhead_target / max_batch_diff), the convergence guard (convergence_guard override + divergence_threshold / no_divergence_guard primitives), partition_ratios, easgd_alpha, meta_controller, and max_overshoot. The builder’s policy/backend are reconciled into elche.mode at build via ElCheMode::from_parts (the inverse of mode.split()). Everything else on DdpRunConfig is run-scope / topology.

checkpoint_every: Option<usize>

Save a checkpoint every N global epochs. None = no checkpointing. Default: None.

snapshot_timeout_secs: u64

Timeout for CPU averaging snapshot collection (seconds). Default: 5. Only applies to AverageBackend::Cpu.

progressive_dispatch: Option<bool>

Enable progressive chunk dispatch for cold-start calibration.

Instead of sending the full epoch partition upfront, the coordinator streams work in small chunks, adapting sizes to measured throughput. This eliminates the idle time on fast GPUs during epoch 0.

Default: None (auto: true for Cadence/Async, false for Sync).

max_grad_norm: Option<f64>

Maximum gradient norm for per-worker clipping.

When set, each worker clips its accumulated gradients (L2 norm) after backward and before the optimizer step. Ensures gradient spikes on any GPU are bounded before they propagate through AllReduce averaging.

vram_pool: bool

Enable the device-resident sample pool on rank workers: leftover VRAM (measured after the first training step) retains samples so later epochs gather them on device instead of re-uploading them. Sizing is automatic; FLODL_VRAM_POOL=off (or 0) in a worker’s env: block is the runtime kill-switch. Default: true.

augment: usize

Augmentation multiplicity: each sample appears k times per epoch in the shared shuffle (pick space len()*k). Pure scheduling — data variation comes from transform, keyed per pick. Default: 1.

epoch_splits: usize

How finely one data pass is sliced into epochs.

At the default 1, an epoch is a full pass over the data. Above 1, an epoch becomes a slice of a pass: num_epochs * epoch_splits epochs run in total, and each sample is still seen exactly num_epochs times. Everything that keys off the epoch boundary follows — eval cadence, checkpointing, and the reduce window, which the coordinator caps at one epoch.

This is what makes a single-pass run (epochs: 1, the normal regime for LLM pretraining) usable: without it such a run has no interior boundary, so no checkpoint and no eval until teardown. Default: 1.

transform: Option<TransformFn>

Deterministic delivery transform applied on each rank, keyed by crate::data::PickKey — the sanctioned augmentation seam. Runs at the worker’s delivery point on freshly assembled rows; the staging tiers retain raw samples only. Default: None.

vram_max_usage: f64

Fraction of total VRAM each rank worker may use for its data plane (prefetch channel + device sample pool), clamped to [0.50, 0.99] at the sizing sites. Default: 0.90 (the solo loader’s vram_max_usage default).

ram_max_usage: f64

Fraction of currently available host RAM each rank’s staging tiers may retain (co-hosted ranks split it consumption- proportionally); the delivery reader ring sizes from the same share. 0.0 disables staging retention and the ring (single- stage fetch). Default: 0.50 (the solo loader’s ram_max_usage default).

gpu_ram_share: Option<f64>

Fraction of physical host RAM (MemTotal) to hand the GPU on an integrated (APU) target, where device memory is carved out of system RAM rather than being a pool of its own — so the host staging tiers and the VRAM pool otherwise price the same DRAM twice and over-commit it.

None (default) reserves whatever aperture the device reports. Ignored on discrete GPUs, where the two pools are genuinely separate. Values above 1.0 are allowed and meaningful: if a platform under-reports MemTotal relative to what the APU can address, a share above 1.0 is how you still express the true reservation. Same knob as DataLoaderBuilder::gpu_ram_share on the solo path.

sample_cache: bool

Pinned RAM sample retention in each rank’s staging tier (see crate::distributed::TrainerConfig::sample_cache). false pins the read-through cache’s budget to zero; the flow window keeps the whole staging share. Default: true.

disk_stage_gb: u64

Local-disk overflow tier under each rank’s sample cache, in GB (see crate::distributed::TrainerConfig::disk_stage_gb). Default: 0 (off).

disk_stage_dir: Option<PathBuf>

Directory for the disk-stage pack file (default: system temp dir).

timeline: Option<Arc<Timeline>>

Optional high-frequency system timeline for profiling DDP behavior.

When set, the coordinator and workers inject training events (sync, epoch boundaries, anchor changes, throttle) into the timeline.

lr_scale_ratio: f64

LR scaling ratio for multi-GPU training. Default: 1.0.

Controls how much the learning rate is scaled with world_size. Formula: lr_factor = 1.0 + (world_size - 1) * ratio.

• 1.0 (default): full linear scaling (Goyal et al., 2017). With 2 GPUs, LR is doubled. Compensates for the LR schedule advancing faster when global_step counts all GPUs’ batches.
• 0.0: no scaling. Each GPU uses the base LR as-is.
• 0.5: half linear scaling. With 2 GPUs, LR *= 1.5.

Tune this if convergence degrades at higher GPU counts.

save_path: Option<String>

Checkpoint bundle stem for the cluster-mode save-on-unrecoverable-failure path. When set on a cluster run, workers persist a <save_path>.fdl / .optim / .meta.json bundle on ShutdownWithSave receipt; see crate::distributed::CheckpointBundle. Required for the via-coord cluster orchestrator entry; optional for non-cluster builds.

Multi-host: this stem must resolve to shared storage. The bundle is split across hosts and each piece is written on its writer’s host: each surviving worker writes its .fdl / .optim on its own host, while the controller writes the .meta.json sidecar (and reads it back on Self::resume_from) on ITS host. A host-local path scatters the bundle and breaks resume. On a genuine multi-host launch the framework prints a one-time reminder; single-box multi-GPU is unaffected (one host owns every piece).

max_failure: Option<MaxFailureThreshold>

Threshold for declaring a cluster run unrecoverable. When the dead-rank count reaches this limit, the coord broadcasts ShutdownWithSave to all survivors. None = no user-configured threshold; backend hard limits still apply (NCCL needs 2+ survivors; CPU needs at least 1). Only honored on cluster-mode runs; non-cluster builds ignore this field.

heartbeat_timeout_secs: Option<u64>

Cluster-mode heartbeat staleness threshold (seconds). If a rank’s last TimingMsg frame is older than this, the controller declares the rank dead and triggers the elastic-membership / max_failure flow. None = use the controller’s built-in default (currently 30s). Only honored on cluster-mode via_coord runs.

epoch_callback_policy: EpochCallbackPolicy

Which rank fires user-supplied per-epoch callbacks (epoch_fn, checkpoint_fn, eval_fn). See EpochCallbackPolicy for the variants. Default EpochCallbackPolicy::Fastest.

eval_every_epochs: Option<usize>

Cadence (in epochs) for the user-supplied eval_fn. Some(n) triggers an eval dispatch every n epochs from the controller’s dispatch_epoch. None or 0 disables. Builder sugar: DdpBuilder::eval_every (accepts EvalCadence).

reports_per_epoch: Option<usize>

Sub-epoch monitor reports per epoch. Some(x) makes the controller emit x per-window metric reports across each epoch (at reduce boundaries), filling the gap the per-epoch feed leaves — decisive for single-epoch LLM training, where the per-epoch curve is one point. None or 0 disables. Builder sugar: DdpBuilder::reports_per_epoch.

record_log_dir: Option<String>

Directory for the persisted monitor record stream. Some(dir) appends every emitted record as JSONL under a node tree mirroring its path (root.log, root/<host>/rank0.log, …); None (the default) keeps the stream live-only. Builder sugar: DdpBuilder::record_log.

max_log_size: Option<u64>

Per-node byte cap for the persisted record stream: each node’s log is a drop-oldest ring bounded to this many bytes, so a long run can never fill the disk. None = DEFAULT_MAX_LOG_BYTES. A tree of N nodes bounds to N * max_log_size.

dashboard_html: Option<String>

Where to write the self-contained dashboard archive at teardown, or None for live-only. One HTML file carrying the epoch feed, the record plane, and the graph SVG — the run’s dashboard as an attachable artifact.

dashboard_theme: Option<String>

Theme the saved archive opens with: None (default) leaves it to the reader’s prefers-color-scheme; "light" / "dark" pin it. "light" is the publication setting — a figure in a paper wants to be light on a reviewer’s dark laptop.

scalar_reductions: Reductions

How each non-core metric key rolls up across ranks in the record tree. Core keys (loss, throughput, batch_share, data_starve, compute_only_ms) are authoritative and cannot be overridden; anything else — every user scalar — defaults to Reduction::Mean, which is wrong for a count (tokens seen, samples processed) and for an extremum (peak memory). Declaring it here both fixes the roll-up and reaches every consumer, because the declarations ride the record stream’s meta record.

resume_from: Option<String>

Checkpoint bundle stem for resume. When set, the cluster orchestrator reads <stem>.meta.json at .run() time, seeds the controller with the saved trajectory state (epoch, global_step, sync_round, ElChe state including TrendGuard history), and kicks the launcher off at meta.epoch instead of 0.

Model parameters and optimizer state are NOT auto-loaded from the bundle by this field — the user’s model_factory / optim_factory closures are the right place for that (call crate::nn::load_checkpoint_file / crate::nn::optim::Stateful::load_state_file inside them). This field carries the controller-side trajectory only.

Multi-host: like Self::save_path, this stem must resolve to shared storage visible to every host (the controller reads the meta on its host; ranks re-seed from the same stem on theirs).

None = fresh run. Builder sugar: DdpBuilder::resume_from.

checkpoint_at_epoch: Option<usize>

Arm a one-shot coverage-granular checkpoint at the first reduce where the cohort reaches this epoch. Progressive modes only (Cadence / Async — a Sync run has no chunk pools to snapshot). Pairs with Self::save_path for the bundle stem: the forged consensus model lands in <stem>.fdl and the trajectory + data-coverage in <stem>.meta.json. None = no mid-run checkpoint. Builder sugar: DdpBuilder::checkpoint_at_epoch.

Implementations

impl DdpRunConfig

pub fn new() -> Self

Create a default config (all defaults).

pub fn with_resume_from(self, stem: impl Into<String>) -> Self

Resume a cluster run from a previously-saved checkpoint bundle.

stem is the path stem used at save time (the value passed to Self::with_save_path / DdpBuilder::save_path). The orchestrator reads <stem>.meta.json at .run() time and seeds the controller with the saved trajectory state. See Self::resume_from for details on what is and isn’t restored.

pub fn with_checkpoint_at_epoch(self, epoch: usize) -> Self

Arm a one-shot coverage-granular checkpoint at the given epoch. Pairs with Self::with_save_path. See Self::checkpoint_at_epoch.

pub fn with_epoch_callback_policy(self, policy: EpochCallbackPolicy) -> Self

Override which rank fires user-supplied per-epoch callbacks. See EpochCallbackPolicy. Default is Fastest.

pub fn with_heartbeat_timeout_secs(self, secs: u64) -> Self

Set the cluster-mode heartbeat staleness threshold (seconds). See Self::heartbeat_timeout_secs.

pub fn with_save_path(self, path: impl Into<String>) -> Self

Set the checkpoint bundle stem for cluster-mode unrecoverable- failure persistence. See crate::distributed::CheckpointBundle for the layout.

pub fn with_max_failure(self, threshold: MaxFailureThreshold) -> Self

Set the unrecoverable-failure threshold for cluster mode.

pub fn with_overhead_target(self, target: f64) -> Self

Set the AllReduce overhead target (fraction of compute time).

pub fn with_max_anchor(self, max: usize) -> Self

Set the maximum anchor count.

pub fn with_min_anchor(self, min: usize) -> Self

Set the minimum anchor count (auto-tune floor).

Forces the overhead auto-tune above its natural equilibrium. Combined with with_max_anchor(min) (same value), pins the anchor at a fixed cadence — useful for fixed-k experiments. The convergence guard and divergence nudge-down paths BYPASS this floor; pair with with_convergence_guard(NoGuard) + with_no_divergence_guard() for truly hard pinning.

pub fn with_anchor(self, anchor: usize) -> Self

Set the initial anchor count.

pub fn with_divergence_threshold(self, threshold: f64) -> Self

Set the divergence threshold for the trend guardrail.

pub fn with_no_divergence_guard(self) -> Self

Disable the divergence guardrail. ElChe’s overhead auto-tune handles cadence alone. Use when you know your workload is stable.

pub fn with_max_batch_diff(self, max: usize) -> Self

Set the maximum batch lead of fastest over slowest worker.

0 = strict lockstep (sync DDP behavior). Workers that exceed this lead are paused until the slowest catches up.

pub fn with_max_overshoot(self, max: usize) -> Self

Set the maximum overshoot past the planned sync point.

Controls cross-epoch streaming aggressiveness. When a GPU finishes its epoch partition, it may stream into the next epoch’s data up to this many batches past ElChe’s planned sync count.

0 disables cross-epoch streaming. Default: auto-tuned.

pub fn with_checkpoint_every(self, n: usize) -> Self

Save a checkpoint every N global epochs.

Requires a checkpoint_fn to be set on the builder. Errors from the checkpoint function are logged but do not stop training.

pub fn with_eval_every_epochs(self, n: usize) -> Self

Fire the user-supplied eval_fn every n epochs from the controller’s dispatch_epoch. n == 0 disables. Builder sugar DdpBuilder::eval_every takes the EvalCadence enum and forwards the integer here.

pub fn with_reports_per_epoch(self, n: usize) -> Self

Emit n sub-epoch monitor reports per epoch (at reduce boundaries). n == 0 disables. Builder sugar: DdpBuilder::reports_per_epoch.

pub fn with_record_log(self, dir: impl Into<String>, max_bytes: u64) -> Self

Persist the monitor record stream as JSONL under dir, each node capped at max_bytes (drop-oldest). max_bytes of 0 uses DEFAULT_MAX_LOG_BYTES.

pub fn with_dashboard_html(self, path: impl Into<String>) -> Self

Save the self-contained dashboard archive to path at teardown.

pub fn with_dashboard_theme(self, theme: impl Into<String>) -> Self

Pin the saved archive’s theme ("light", "dark", or "auto").

pub fn with_scalar_reduction( self, key: impl Into<String>, reduction: Reduction, ) -> Self

Declare how one non-core metric key rolls up across ranks. Repeatable; a later declaration for the same key replaces the earlier one. Core keys are authoritative and silently keep their own reduction.

pub fn with_snapshot_timeout(self, secs: u64) -> Self

Set the timeout for CPU averaging snapshot collection (seconds).

Default: 5. Only applies to AverageBackend::Cpu. If not all worker snapshots arrive within this timeout, the averaging attempt is aborted and retried on the next cycle.

pub fn with_partition_ratios(self, ratios: &[f64]) -> Self

Set explicit per-rank partition ratios (e.g. &[0.7, 0.3]).

Disables automatic throughput-based rebalancing. Ratios are normalized so they sum to 1.0. Length must match world_size at launch time.

pub fn with_progressive_dispatch(self, enabled: bool) -> Self

Enable or disable progressive chunk dispatch.

When enabled, the coordinator streams work in small chunks instead of sending full epoch partitions. This allows continuous throughput adaptation and eliminates cold-start idle time.

Default: auto (true for Cadence/Async, false for Sync).

pub fn with_max_grad_norm(self, max_norm: f64) -> Self

Set maximum gradient norm for per-worker clipping.

Each worker clips accumulated gradients to this L2 norm after backward and before the optimizer step. Prevents gradient spikes on any GPU from propagating through AllReduce.

pub fn with_vram_pool(self, enabled: bool) -> Self

Enable / disable the device-resident sample pool on rank workers (see Self::vram_pool). Default: enabled.

pub fn with_augment(self, k: usize) -> Self

Augmentation multiplicity (see Self::augment).

pub fn with_epoch_splits(self, n: usize) -> Self

Slices per data pass (see Self::epoch_splits).

pub fn with_transform( self, f: impl Fn(Vec<Tensor>, &[PickKey]) -> Result<Vec<Tensor>> + Send + Sync + 'static, ) -> Self

Delivery transform (see Self::transform).

pub fn with_vram_max_usage(self, max_usage: f64) -> Self

VRAM share for each rank’s data plane (see Self::vram_max_usage).

pub fn with_ram_max_usage(self, max_usage: f64) -> Self

Host-RAM share for each rank’s staging tiers (see Self::ram_max_usage).

pub fn with_gpu_ram_share(self, share: f64) -> Self

Fraction of physical host RAM (MemTotal) reserved for the GPU on an integrated (APU) target (see Self::gpu_ram_share). Ignored on discrete GPUs. Same knob as DataLoaderBuilder::gpu_ram_share on the solo path.

pub fn with_sample_cache(self, enabled: bool) -> Self

Pinned RAM sample retention (see Self::sample_cache).

pub fn with_disk_stage(self, gb: u64) -> Self

Local-disk overflow tier in GB (see Self::disk_stage_gb).

pub fn with_disk_stage_dir(self, dir: impl Into<PathBuf>) -> Self

Disk-stage directory (see Self::disk_stage_dir).

pub fn with_timeline(self, tl: Arc<Timeline>) -> Self

Attach a high-frequency system timeline for profiling DDP behavior.

When set, the coordinator and workers inject training events (sync, epoch, anchor changes, throttle) into the timeline.

pub fn with_lr_scale_ratio(self, ratio: f64) -> Self

Set the LR scaling ratio for multi-GPU training.

Formula: lr_factor = 1.0 + (world_size - 1) * ratio. Default: 1.0 (full linear scaling). Set to 0.0 to disable.

pub fn with_elche_relax_up(self, enabled: bool) -> Self

Allow or suppress ElChe’s anchor relax-up on stable convergence.

Default: false (off). Set to true to enable: each Stable convergence-guard verdict will grow the anchor via el_che.relax_anchor_up(). Opt in when measuring the relax-up regime; the default keeps the anchor under overhead-based control alone, matching pre-relax-up behavior.

pub fn with_easgd_alpha(self, alpha: f64) -> Self

Set the EASGD elastic averaging weight α. Must be in (0, 1]. None (default) is full overwrite (equivalent to α=1.0 with the fast copy_ path). Values in (0, 1) enable elastic blending on the cpu-async path; α=1.0 also enables blending but is functionally identical to the overwrite default. See easgd_alpha field docs for the formula and reference.

pub fn with_meta_controller(self, enabled: bool) -> Self

Enable the LR-aware meta-controller above ElChe.

On by default. See the meta_controller field for behavior and crate::distributed::lr_event_meta for the design.

Trait Implementations

impl Clone for DdpRunConfig

fn clone(&self) -> DdpRunConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DdpRunConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DdpRunConfig

fn default() -> Self

Returns the “default value” for a type.