Skip to main content

baseRT_suggest_max_context_multi

Function baseRT_suggest_max_context_multi 

Source
pub unsafe extern "C" fn baseRT_suggest_max_context_multi(
    model_paths: *const *const c_char,
    n_models: c_int,
    max_batch: c_int,
    kv_bits: c_int,
    paged_kv: c_int,
) -> c_int
Expand description

As above for a set of models that will be resident at the same time: weights and KV pools add up, and the trained-window cap is the shortest among them.