Struct RuntimeConfig 

pub struct RuntimeConfig {

    pub model_path: PathBuf,
    pub format: ModelFormat,
    pub host: String,
    pub port: u16,
    pub context_size: u32,
    pub batch_size: u32,
    pub threads: u32,
    pub gpu_layers: u32,
    pub parallel_slots: u32,
    pub ubatch_size: u32,
    pub runtime_binary: Option<PathBuf>,
    pub draft_model_path: Option<PathBuf>,
    pub speculative_draft_max: u32,
    pub speculative_draft_p_min: f32,
    pub extra_config: Value,
}

Runtime configuration for model initialization

Fields

model_path: PathBuf

Path to model file

format: ModelFormat

Model format

host: String

Host for runtime server (e.g., “127.0.0.1”)

port: u16

Port for runtime server (e.g., 8001)

context_size: u32

Context size

batch_size: u32

Batch size

threads: u32

Number of CPU threads

gpu_layers: u32

GPU layers to offload (0 = CPU only)

parallel_slots: u32

Number of parallel KV-cache slots (continuous batching slots). Should match MAX_CONCURRENT_STREAMS. Maps to llama-server –parallel N.

ubatch_size: u32

Micro-batch size for GPU compute. Larger values increase tensor-core utilisation. Maps to llama-server –ubatch-size N.

runtime_binary: Option<PathBuf>

Path to runtime binary (e.g., llama-server.exe)

draft_model_path: Option<PathBuf>

Path to draft model for speculative decoding. None = disabled. Maps to llama-server –model-draft.

speculative_draft_max: u32

Maximum draft tokens generated per speculative step. Maps to llama-server –draft-max. Default: 8.

speculative_draft_p_min: f32

Minimum acceptance probability for a draft token. Maps to llama-server –draft-p-min. Default: 0.4.

extra_config: Value

Additional runtime-specific configuration

Trait Implementations

impl Clone for RuntimeConfig

fn clone(&self) -> RuntimeConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for RuntimeConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RuntimeConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for RuntimeConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for RuntimeConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.