Enum ServeCommands 

pub enum ServeCommands {
    Plan {
        model: String,
        gpu: bool,
        batch_size: usize,
        seq_len: usize,
        format: String,
        quant: Option<String>,
    },
    Run {

        file: PathBuf,
        port: u16,
        host: String,
        no_cors: bool,
        no_metrics: bool,
        no_gpu: bool,
        gpu: bool,
        batch: bool,
        trace: bool,
        trace_level: String,
        profile: bool,
        backend: Option<String>,
        otlp_endpoint: Option<String>,
        context_length: usize,
        no_fp8_cache: bool,
    
},
}

Inference server subcommands (plan/run).

apr serve plan computes VRAM budget, throughput estimates, and contract verification before starting a server. apr serve run launches the server.

Variants

Plan

Pre-flight inference capacity plan (VRAM budget, roofline, contracts)

Inspects model metadata, detects GPU hardware, and produces a capacity plan showing whether the model fits in VRAM with the requested batch size. No weights are loaded — header-only inspection.

Accepts local files (.gguf, .apr, .safetensors) or HuggingFace repo IDs (hf://org/repo or org/repo). For HF repos, only the ~2KB config.json is fetched — no weight download needed.

Fields

model: String

Model source: local path or HuggingFace repo (hf://org/repo, org/repo)

gpu: bool

Detect GPU via nvidia-smi for VRAM budget

batch_size: usize

Target batch size for throughput estimation

seq_len: usize

Sequence length for KV cache estimation

format: String

Output format: text, json, yaml

quant: Option<String>

Quantization override for HF models (e.g., Q4_K_M, Q6_K, F16)

Run

Start inference server (REST API, streaming, metrics)

Fields

file: PathBuf

Path to model file

port: u16

Port to listen on

host: String

Host to bind to

no_cors: bool

Disable CORS

no_metrics: bool

Disable Prometheus metrics endpoint

no_gpu: bool

Disable GPU acceleration

gpu: bool

Force GPU acceleration (requires CUDA)

batch: bool

Enable batched GPU inference for 2X+ throughput

trace: bool

Enable inference tracing (PMAT-SHOWCASE-METHODOLOGY-001)

trace_level: String

Trace detail level (none, basic, layer)

profile: bool

Enable inline Roofline profiling (adds X-Profile headers)

backend: Option<String>

PMAT-332: Compute backend override (cuda, cpu, wgpu)

otlp_endpoint: Option<String>

PMAT-485: OTLP endpoint for distributed tracing export (Jaeger/Tempo)

When set, inference spans (W3C Trace Context) are exported via OTLP. Each request = parent span, each layer = child span with TensorStats. Example: –otlp-endpoint http://localhost:4317

context_length: usize

GH-286: Max context/sequence length for KV cache. Lower = less RSS.

no_fp8_cache: bool

GH-286: Skip FP8 weight cache warmup. Saves ~1.5 GB RSS.

Trait Implementations

impl Debug for ServeCommands

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromArgMatches for ServeCommands

fn from_arg_matches(__clap_arg_matches: &ArgMatches) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn from_arg_matches_mut( __clap_arg_matches: &mut ArgMatches, ) -> Result<Self, Error>

Instantiate Self from ArgMatches, parsing the arguments as needed.

fn update_from_arg_matches( &mut self, __clap_arg_matches: &ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

fn update_from_arg_matches_mut<'b>( &mut self, __clap_arg_matches: &mut ArgMatches, ) -> Result<(), Error>

Assign values from ArgMatches to self.

impl Subcommand for ServeCommands

fn augment_subcommands<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate Self via FromArgMatches::from_arg_matches_mut

fn augment_subcommands_for_update<'b>(__clap_app: Command) -> Command

Append to Command so it can instantiate self via FromArgMatches::update_from_arg_matches_mut

fn has_subcommand(__clap_name: &str) -> bool

Test whether Self can parse a specific subcommand