Skip to main content

baseRT_suggest_max_context_spec

Function baseRT_suggest_max_context_spec 

Source
pub unsafe extern "C" fn baseRT_suggest_max_context_spec(
    model_paths: *const *const c_char,
    n_models: c_int,
    speculator_paths: *const *const c_char,
    speculator_embedded: *const c_int,
    speculator_target: *const c_int,
    n_speculators: c_int,
    max_batch: c_int,
    kv_bits: c_int,
    paged_kv: c_int,
) -> c_int
Expand description

As above, plus the speculators loading beside those models: speculator_embedded[i] non-zero is the target bundle’s embedded head (its KV pool only); zero (or a null array) a sidecar / draft model (weights and KV pool), even when also served. speculator_target[i]

= 0 is the index of the model a sidecar’s baseRT_load_drafter reopens for its embedding / lm_head (charged where the backend copies wrapped weights); -1 (or a null array) for none. Speculators never cap the window.