Enum TrainCommands 

pub enum TrainCommands {
    Plan {

        data: Option<PathBuf>,
        model_size: String,
        model_path: Option<PathBuf>,
        num_classes: usize,
        task: String,
        config: Option<PathBuf>,
        output: PathBuf,
        strategy: String,
        budget: usize,
        scout: bool,
        max_epochs: usize,
        learning_rate: Option<f32>,
        lora_rank: Option<usize>,
        batch_size: Option<usize>,
        val_data: Option<PathBuf>,
        test_data: Option<PathBuf>,
        format: String,
    
},
    Apply {

        plan: Option<PathBuf>,
        config: Option<PathBuf>,
        task: String,
        data: Option<PathBuf>,
        model_size: String,
        model_path: Option<PathBuf>,
        num_classes: usize,
        output: PathBuf,
        strategy: String,
        budget: usize,
        scout: bool,
        max_epochs: usize,
        learning_rate: Option<f32>,
        lora_rank: Option<usize>,
        batch_size: Option<usize>,
        distributed: bool,
        world_size: Option<usize>,
        rank: Option<usize>,
        coordinator_addr: Option<String>,
        deterministic: bool,
        seed: Option<u64>,
    
},
    Watch {
        config: PathBuf,
        max_restarts: usize,
        heartbeat_timeout: u64,
        backoff_initial: u64,
        backoff_max: u64,
    },
    Sweep {
        config: PathBuf,
        strategy: String,
        num_configs: usize,
        output_dir: PathBuf,
        seed: u64,
    },
    Archive {
        checkpoint_dir: PathBuf,
        output: PathBuf,
        version: Option<String>,
        notes: Option<String>,
    },
    Submit {
        cluster: PathBuf,
        model: PathBuf,
        adapters: Vec<String>,
        rank: u32,
        epochs: u32,
        budget_mb: u64,
        dry_run: bool,
    },
    ClusterStatus {
        cluster: PathBuf,
    },
}

Training pipeline subcommands (forjar-style plan/apply).

Thin CLI wrappers around entrenar’s training plan/apply infrastructure.

Variants

Plan

Generate a training plan without touching the GPU.

Validates data quality, checks model compatibility, builds HPO search space, estimates resource usage, and runs pre-flight checks. Outputs a serializable plan manifest (text, JSON, or YAML).

Analogous to forjar plan — shows what will happen before committing GPU time.

Fields

data: Option<PathBuf>

Path to training data (JSONL) — required for –task classify

model_size: String

Model size: “0.5B”, “9B”, “7B”, “13B”

model_path: Option<PathBuf>

Path to model weights directory

num_classes: usize

Number of output classes

task: String

Task type: classify, pretrain

config: Option<PathBuf>

YAML training config (for –task pretrain)

output: PathBuf

Output directory for checkpoints

strategy: String

HPO strategy: tpe, grid, random, manual

budget: usize

HPO budget (number of trials)

scout: bool

Scout mode: 1 epoch per trial for fast exploration

max_epochs: usize

Maximum epochs per trial

learning_rate: Option<f32>

Manual learning rate (only used with –strategy manual)

lora_rank: Option<usize>

Manual LoRA rank (only used with –strategy manual)

batch_size: Option<usize>

Manual batch size (only used with –strategy manual)

val_data: Option<PathBuf>

Validation data file (JSONL)

test_data: Option<PathBuf>

Test data file (JSONL)

format: String

Output format: text, json, yaml

Apply

Execute a training plan (allocate GPU, run trials).

Reads a previously generated plan (YAML/JSON) and executes it:

• Manual strategy: single training run with specified hyperparameters
• HPO strategy: multiple trials with automatic hyperparameter tuning

Analogous to forjar apply — commits resources and executes the plan.

Fields

plan: Option<PathBuf>

Path to a saved plan file (YAML or JSON from apr train plan)

config: Option<PathBuf>

YAML training config (for –task pretrain)

task: String

Task type: classify, pretrain

data: Option<PathBuf>

Path to training data (JSONL)

model_size: String

Model size: “0.5B”, “9B”, “7B”, “13B”

model_path: Option<PathBuf>

Path to model weights directory

num_classes: usize

Number of output classes

output: PathBuf

Output directory for checkpoints and leaderboard

strategy: String

HPO strategy: tpe, grid, random, manual

budget: usize

HPO budget (number of trials)

scout: bool

Scout mode: 1 epoch per trial

max_epochs: usize

Maximum epochs per trial

learning_rate: Option<f32>

Manual learning rate (only used with –strategy manual)

lora_rank: Option<usize>

Manual LoRA rank (only used with –strategy manual)

batch_size: Option<usize>

Manual batch size (only used with –strategy manual)

distributed: bool

Enable distributed data-parallel training

world_size: Option<usize>

Total number of workers (default: auto-detect GPUs)

rank: Option<usize>

This worker’s global rank (default: 0 = coordinator)

coordinator_addr: Option<String>

Coordinator address for distributed training (default: 0.0.0.0:9000)

deterministic: bool

Enable bitwise deterministic training (CUBLAS_WORKSPACE_CONFIG, cuDNN deterministic)

seed: Option<u64>

Random seed for reproducibility (default: from YAML or 42)

Watch

Watch a training run with automatic restart on crash and hang detection.

Monitors a running or to-be-started training process:

• Detects crashes (SIGABRT, SIGSEGV, OOM) and restarts with backoff
• Detects hangs via heartbeat/training_state.json staleness
• Captures GPU state and crash diagnostics
• Auto-enables CUDA_LAUNCH_BLOCKING on async crash pattern

Sovereign Rust replacement for train-guard.sh.

Fields

config: PathBuf

YAML training config to run and watch

max_restarts: usize

Maximum number of restart attempts

heartbeat_timeout: u64

Heartbeat staleness threshold in seconds

backoff_initial: u64

Initial backoff delay in seconds

backoff_max: u64

Maximum backoff delay in seconds

Sweep

Generate hyperparameter sweep configs from a base YAML.

Creates N training configs with varied hyperparameters using grid or random search. Each config is a complete YAML that can be passed to apr train apply --task pretrain --config <file>.

Sovereign Rust replacement for hyperparam-sweep.py.

Fields

config: PathBuf

Base YAML training config to sweep from

strategy: String

Search strategy: grid or random

num_configs: usize

Number of configs to generate (random) or max combinations (grid)

output_dir: PathBuf

Output directory for generated configs

seed: u64

Seed for random search reproducibility

Archive

Archive a checkpoint into a release bundle.

Packages model weights, config, training state, and metadata into a self-contained directory with integrity manifest.

Fields

checkpoint_dir: PathBuf

Path to checkpoint directory

output: PathBuf

Output archive directory

version: Option<String>

Release version tag (e.g., “v1.0”)

notes: Option<String>

Release notes

Submit

Submit multi-adapter training jobs to a cluster (GPU-SHARE Phase 3).

Reads a cluster.yaml config, places adapter jobs across nodes using the greedy placement algorithm, and generates launch commands.

Fields

cluster: PathBuf

Path to cluster config YAML

model: PathBuf

Model checkpoint path (.apr)

adapters: Vec<String>

Adapter specs: DATA:CHECKPOINT pairs (one per adapter)

rank: u32

LoRA rank

epochs: u32

Number of training epochs

budget_mb: u64

Estimated VRAM budget per adapter (MB)

dry_run: bool

Dry run: show placement and commands without executing

ClusterStatus

Show cluster status: nodes, GPUs, adapter capacity (GPU-SHARE Phase 3).

Reads a cluster.yaml config and displays node health, VRAM availability, and adapter placement capacity.

Fields

cluster: PathBuf

Path to cluster config YAML

Trait Implementations

impl Debug for TrainCommands

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromArgMatches for TrainCommands

fn from_arg_matches(__clap_arg_matches: &ArgMatches) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn from_arg_matches_mut( __clap_arg_matches: &mut ArgMatches, ) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn update_from_arg_matches( &mut self, __clap_arg_matches: &ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

fn update_from_arg_matches_mut<'b>( &mut self, __clap_arg_matches: &mut ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

impl Subcommand for TrainCommands

fn augment_subcommands<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate Self via FromArgMatches::from_arg_matches_mut

fn augment_subcommands_for_update<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate self via FromArgMatches::update_from_arg_matches_mut

fn has_subcommand(__clap_name: &str) -> bool

Test whether Self can parse a specific subcommand