pub struct MockEngineArgs {Show 59 fields
pub engine_type: EngineType,
pub num_gpu_blocks: usize,
pub block_size: usize,
pub max_model_len: Option<usize>,
pub max_num_seqs: Option<usize>,
pub max_num_batched_tokens: Option<usize>,
pub enable_prefix_caching: bool,
pub g1_backend: Option<G1Backend>,
pub enable_chunked_prefill: bool,
pub speedup_ratio: f64,
pub decode_speedup_ratio: f64,
pub dp_size: u32,
pub startup_time: Option<f64>,
pub worker_type: WorkerType,
pub planner_profile_data: Option<PathBuf>,
pub perf_model: Arc<PerfModel>,
pub aic_backend: Option<String>,
pub aic_system: Option<String>,
pub aic_backend_version: Option<String>,
pub aic_tp_size: Option<usize>,
pub aic_model_path: Option<String>,
pub aic_moe_tp_size: Option<usize>,
pub aic_moe_ep_size: Option<usize>,
pub aic_attention_dp_size: Option<usize>,
pub aic_gemm_dtype: Option<String>,
pub aic_moe_dtype: Option<String>,
pub aic_fmha_dtype: Option<String>,
pub aic_kv_cache_dtype: Option<String>,
pub aic_comm_dtype: Option<String>,
pub aic_nextn: Option<usize>,
pub aic_nextn_accept_rates: Option<String>,
pub aic_mtp_seed: u64,
pub gpu_memory_utilization: Option<f64>,
pub mem_fraction_static: Option<f64>,
pub free_gpu_memory_fraction: Option<f64>,
pub enable_local_indexer: bool,
pub bootstrap_port: Option<u16>,
pub handoff_session_timeout_ms: u64,
pub kv_bytes_per_token: Option<usize>,
pub kv_transfer_bandwidth: Option<f64>,
pub kv_transfer_timing_mode: KvTransferTimingMode,
pub num_g2_blocks: Option<usize>,
pub num_g3_blocks: Option<usize>,
pub enable_g4_storage: bool,
pub offload_batch_size: Option<usize>,
pub bandwidth_g1_to_g2_gbps: Option<f64>,
pub bandwidth_g2_to_g1_gbps: Option<f64>,
pub bandwidth_g2_to_g3_gbps: Option<f64>,
pub bandwidth_g3_to_g2_gbps: Option<f64>,
pub bandwidth_g2_to_g4_gbps: Option<f64>,
pub bandwidth_g4_to_g2_gbps: Option<f64>,
pub reasoning: Option<ReasoningConfig>,
pub response_replay_trace_path: Option<PathBuf>,
pub zmq_kv_events_port: Option<u16>,
pub zmq_replay_port: Option<u16>,
pub preemption_mode: PreemptionMode,
pub router_queue_policy: Option<RouterQueuePolicy>,
pub sglang: Option<SglangArgs>,
pub trtllm: Option<TrtllmArgs>,
}Expand description
Configuration arguments for MockEngine
Fields§
§engine_type: EngineTypeEngine type: vLLM, SGLang, or TensorRT-LLM simulation
num_gpu_blocks: usizeUsable simulated G1 capacity. This preserves the mocker’s historical
convention across backends. A raw vLLM num_gpu_blocks value also
includes its reserved null block, so parity runs configure real vLLM
with one additional total block.
block_size: usize§max_model_len: Option<usize>Optional vLLM sequence-length limit, including prompt and generated tokens. Requests with no room to generate are rejected before admission.
max_num_seqs: Option<usize>§max_num_batched_tokens: Option<usize>§enable_prefix_caching: bool§g1_backend: Option<G1Backend>Requested G1 block-manager implementation for the shared vLLM/TRT-LLM
scheduler. None selects native unless legacy offload requires KVBM.
Ignored by the SGLang scheduler, which uses SglangKvManager.
enable_chunked_prefill: bool§speedup_ratio: f64§decode_speedup_ratio: f64Additional speedup multiplier applied only to decode steps.
Models speculative decoding (e.g. Eagle) where decode throughput improves
without affecting prefill latency. The effective decode speedup is
speedup_ratio * decode_speedup_ratio.
dp_size: u32§startup_time: Option<f64>Optional startup time in seconds to simulate engine initialization delay
worker_type: WorkerTypeWorker type for disaggregated serving (Aggregated, Prefill, or Decode)
planner_profile_data: Option<PathBuf>Original planner profile NPZ path used to materialize perf_model.
perf_model: Arc<PerfModel>Performance model for timing predictions (not serialized, loaded from planner_profile_data)
aic_backend: Option<String>If set, indicates direct AIC SDK calls should be used. The value is the backend name (e.g., “sglang”, “vllm”). The Python layer reads this and overrides perf_model with an Aiconfigurator callback.
aic_system: Option<String>AIC GPU system name (e.g., “h200_sxm”). Required when aic_backend is set.
aic_backend_version: Option<String>AIC backend engine version (e.g., “0.12.0” for vLLM, “0.5.6.post2” for SGLang). If None, uses the default version for the backend.
aic_tp_size: Option<usize>Tensor parallel size for AIC latency prediction. Only affects AIC performance model lookups, not mocker scheduling.
aic_model_path: Option<String>HuggingFace model path for AIC latency prediction (e.g., “nvidia/Llama-3.1-8B-Instruct-FP8”).
aic_moe_tp_size: Option<usize>MoE tensor-parallel size for AIC latency prediction (e.g., 4 for pure MoE-TP). Required for MoE models; must satisfy: aic_tp_size * aic_attention_dp_size == aic_moe_tp_size * aic_moe_ep_size.
aic_moe_ep_size: Option<usize>MoE expert-parallel size for AIC latency prediction (e.g., 4 for pure EP). Required for MoE models; must satisfy: aic_tp_size * aic_attention_dp_size == aic_moe_tp_size * aic_moe_ep_size.
aic_attention_dp_size: Option<usize>Attention data-parallel size for AIC latency prediction (default: 1).
Corresponds to the dp dimension in AIC CLI output.
Must satisfy: aic_tp_size * aic_attention_dp_size == aic_moe_tp_size * aic_moe_ep_size.
aic_gemm_dtype: Option<String>Weight dtype override for AIC latency prediction.
aic_moe_dtype: Option<String>MoE kernel dtype override for AIC latency prediction.
aic_fmha_dtype: Option<String>Activation dtype override for AIC latency prediction.
aic_kv_cache_dtype: Option<String>KV-cache dtype override for AIC latency prediction.
aic_comm_dtype: Option<String>Communication (collective) dtype override for AIC latency prediction.
aic_nextn: Option<usize>MTP/Eagle speculative-decoding draft-token count (1..=5). The mocker samples accepted drafts while AIC supplies undiscounted verification-round latency.
aic_nextn_accept_rates: Option<String>Conditional acceptance rates for draft tokens, comma-separated. Entry i is P(draft i accepted | every earlier draft was accepted).
aic_mtp_seed: u64Base RNG seed for MTP burst sampling. Worker rank is added with wrapping arithmetic before constructing each worker-local sampler.
gpu_memory_utilization: Option<f64>GPU memory fraction for AIC KV capacity estimation with vLLM.
mem_fraction_static: Option<f64>Static memory fraction for AIC KV capacity estimation with SGLang.
free_gpu_memory_fraction: Option<f64>Fraction of free GPU memory (after weights/buffers) allocated to the KV
cache, for AIC KV capacity estimation with TRT-LLM. Mirrors TRT-LLM’s
KvCacheConfig.free_gpu_memory_fraction. Unlike vLLM’s
gpu_memory_utilization (a fraction of total memory), this is a
fraction of what remains after the model is loaded.
enable_local_indexer: boolEnable worker-local KV indexer for tracking this worker’s own KV cache state
bootstrap_port: Option<u16>Bootstrap port for disaggregated serving rendezvous. Prefill workers listen on this port; decode workers connect to it. If None, bootstrap rendezvous is disabled.
handoff_session_timeout_ms: u64Absolute live handoff session timeout, excluding modeled transfer delay.
kv_bytes_per_token: Option<usize>KV cache bytes per token, auto-computed from model config by Python CLI. Formula: num_layers * 2 * num_kv_heads * head_dim * dtype_bytes
kv_transfer_bandwidth: Option<f64>KV cache transfer bandwidth in GB/s for disaggregated serving latency simulation. Default: 64.0 (inter-node InfiniBand). Set to 0 to disable KV transfer delay. For intra-node NVLink, typical value is ~450.
kv_transfer_timing_mode: KvTransferTimingModeSelects whether disaggregated transfer timing charges the full prompt or only the physical prompt footprint missing at the destination.
num_g2_blocks: Option<usize>KVBM G2 (host DRAM) block capacity. When the kvbm-offload
feature is enabled, setting this explicitly opts the mocker into
G2 offload simulation. When unset or set to 0, no G2 offload engine
is attached.
num_g3_blocks: Option<usize>KVBM G3 shared lower-tier block capacity. Positive values require
num_g2_blocks and a resolvable KV block byte size; 0 disables G3.
enable_g4_storage: boolEnable KVBM mock G4 object-storage simulation. G4 stages through G2
and uses object presence operations instead of a BlockManager<G4>.
offload_batch_size: Option<usize>Batch size for the G1→G2 offload pipeline. Offloads are grouped
into batches of this size before being handed to the worker.
Only consulted when the kvbm-offload feature is enabled;
falls back to the KvbmOffloadConfig default when unset or 0.
bandwidth_g1_to_g2_gbps: Option<f64>G1→G2 offload bandwidth in GB/s for the PS-queue simulation.
Only consulted when the kvbm-offload feature is enabled;
falls back to the KvbmOffloadConfig default (host DRAM PCIe
ballpark) when unset.
bandwidth_g2_to_g1_gbps: Option<f64>G2→G1 onboard bandwidth in GB/s for the PS-queue simulation.
Only consulted when the kvbm-offload feature is enabled;
falls back to the KvbmOffloadConfig default when unset.
bandwidth_g2_to_g3_gbps: Option<f64>G2→G3 offload bandwidth in GB/s for the shared PS-queue simulation.
bandwidth_g3_to_g2_gbps: Option<f64>G3→G2 staging bandwidth in GB/s for the shared PS-queue simulation.
bandwidth_g2_to_g4_gbps: Option<f64>G2→G4 object offload bandwidth in GB/s for the shared PS-queue simulation.
bandwidth_g4_to_g2_gbps: Option<f64>G4→G2 object staging bandwidth in GB/s for the shared PS-queue simulation.
reasoning: Option<ReasoningConfig>Reasoning/thinking token configuration. When set, the mocker wraps output in thinking boundary tokens.
response_replay_trace_path: Option<PathBuf>Optional Mooncake trace with exact output token IDs keyed by
output_replay_id annotations. Direct replay paths carry the same token
IDs on DirectRequest and do not need this lookup.
zmq_kv_events_port: Option<u16>ZMQ port for publishing KV events in vLLM’s native wire format. When set, the scheduler publishes to a ZMQ PUB socket instead of directly to NATS. A KvEventPublisher relay subscribes to this socket and forwards events to NATS.
zmq_replay_port: Option<u16>ZMQ ROUTER port for replay of buffered KV event batches.
When set alongside zmq_kv_events_port, the mocker binds a ROUTER socket
that streams back buffered batches by sequence number on request.
Port is offset by dp_rank (replay_port + dp_rank).
preemption_mode: PreemptionModePreemption mode for decode eviction under memory pressure. Lifo (default) evicts the newest request; Fifo evicts the oldest.
router_queue_policy: Option<RouterQueuePolicy>Optional replay-only override for the router queue policy.
sglang: Option<SglangArgs>SGLang-specific configuration. Only used when engine_type == Sglang.
trtllm: Option<TrtllmArgs>TensorRT-LLM-specific configuration. Only used when engine_type == Trtllm.
Implementations§
Source§impl MockEngineArgs
impl MockEngineArgs
pub fn builder() -> MockEngineArgsBuilder
Sourcepub fn aic_gpus_per_worker(&self) -> usize
pub fn aic_gpus_per_worker(&self) -> usize
GPUs occupied by one worker (engine), derived from tensor parallelism
and the materialized DP topology. AIC-backed replay uses
aic_tp_size × aic_attention_dp_size; non-AIC replay still counts one
GPU for every independently modeled dp_size rank. Used to turn
provisioned worker-seconds into GPU-hours.
Sourcepub fn effective_handoff_capacity(&self) -> usize
pub fn effective_handoff_capacity(&self) -> usize
Finite ownership bound for live handoff queues and sessions.
An unset runnable-sequence limit is semantically unbounded, so use the physical KV block count as the conservative process-local bound.
pub fn normalized(self) -> Result<Self>
Sourcepub fn resolved_g1_backend(&self) -> G1Backend
pub fn resolved_g1_backend(&self) -> G1Backend
Return the selected backend, resolving an unset raw configuration from its engine and lower-tier offload settings.
Sourcepub fn scheduling_policy(&self) -> SchedulingPolicy
pub fn scheduling_policy(&self) -> SchedulingPolicy
Scheduling policy applied by the shared vLLM scheduler core, derived
from the engine type. TRT-LLM uses GUARANTEED_NO_EVICT.
pub fn is_prefill(&self) -> bool
pub fn is_decode(&self) -> bool
pub fn needs_kv_publisher(&self) -> bool
pub fn undiscounted_aic_accept_rates(&self) -> Option<String>
Sourcepub fn from_json_file(path: &Path) -> Result<Self>
pub fn from_json_file(path: &Path) -> Result<Self>
Create MockEngineArgs from a JSON file containing extra engine arguments
pub fn from_json_str(content: &str) -> Result<Self>
Trait Implementations§
Source§impl Clone for MockEngineArgs
impl Clone for MockEngineArgs
Source§fn clone(&self) -> MockEngineArgs
fn clone(&self) -> MockEngineArgs
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more