Struct RecommendedSampling 

pub struct RecommendedSampling {
    pub temperature: Option<f32>,
    pub top_p: Option<f32>,
    pub top_k: Option<usize>,
}

The sampling a checkpoint recommends for itself, one Option per field so that “this model says nothing about top_p” stays distinguishable from “this model recommends top_p = 1.0”. This is sglang’s sampling_defaults='model', ported from FreeToken python/freetoken/utils/hf.py:92 load_generation_sampling.

Every field is None for a checkpoint that recommends nothing, which is the overwhelming majority, and RecommendedSampling::resolve then reproduces ferrox’s existing defaults exactly – a recommendation may only fill a gap the request left, never override it.

Why this exists at all: reasoning checkpoints are tuned for a specific sampler (Qwen3.5 ships temperature 1.0, top_k 20, top_p 0.95) and ship those numbers with the weights. Served under a generic greedy-or-0.8 default they fall into repetition loops – fluent output that never terminates – which reads as a broken model rather than as a serving default nobody read off the file.

Fields

temperature: Option<f32>

top_p: Option<f32>

top_k: Option<usize>

Implementations

impl RecommendedSampling

pub fn from_gguf(file: &impl TensorSource) -> Self

The sampling a GGUF recommends for itself, from the general.sampling.* metadata keys llama.cpp’s converter writes when the source checkpoint carried a generation_config.json.

This is the GGUF half of FreeToken’s load_generation_sampling (python/freetoken/utils/hf.py:92), which checks the GGUF metadata first and only falls back to a generation_config.json sidecar for non-GGUF checkpoints – a GGUF is a single file and has no sidecar to read.

Key names are llama.cpp’s own (general.sampling.temp, not temperature). Each key is independent: a file that names only top_k recommends only top_k, and the two fields it did not mention stay None so the server’s own defaults keep speaking for them.

temp / top_p are read as float or integer, because a converter that wrote temp = 1 stores a GGUF integer and dropping that value would silently serve the checkpoint greedy – the exact repetition-loop failure the recommendation exists to prevent.

impl RecommendedSampling

pub fn is_empty(&self) -> bool

True when the checkpoint recommended nothing at all, i.e. Self::resolve is guaranteed to return the framework defaults for any request. Useful for telling an operator whether “model defaults” had anything to act on.

pub fn resolve( &self, requested: RequestedSampling, framework: SamplingParams, ) -> SamplingParams

Precedence, exactly as FreeToken’s resolve_sampling.pick (python/freetoken/server/generation.py:170) applies it: the request’s own value, else the checkpoint’s recommendation, else the framework default carried by framework (ferrox’s SamplingParams::default() unless a caller has its own).

The penalty fields are taken from framework untouched: nothing in the reference reads a recommended penalty, and inventing one here would be this function changing generation on its own.

Getting the order wrong in either direction is a silent behaviour change: recommendation-over-request makes a client’s explicit temperature: 0 unreachable on a model that recommends 1.0, and framework-over-recommendation is the greedy repetition loop this whole path exists to avoid.

pub fn from_generation_config(json: &str) -> Self

The recommendation in a HuggingFace-style generation_config.json body.

Two rules, both from the reference (hf.py:92 load_generation_sampling):

• do_sample: false means the checkpoint recommends greedy, which is returned as temperature = 0 and nothing else – the top_k/top_p in such a file describe a sampler the model asks not to be used.
• otherwise only the keys actually present are returned. An absent key stays None; filling it with a house default (the naive reading, and what HF’s own GenerationConfig object does for you) would turn silence into a recommendation and let a file that says only temperature: 0.6 also pin top_p to 1.0, overriding the server’s own default for a value the checkpoint never expressed.

A file that does not parse, or is not a JSON object, recommends nothing – a malformed sidecar must not be able to change how a model is sampled.

pub fn from_model_dir(dir: &Path) -> Self

Self::from_generation_config for the generation_config.json beside a checkpoint’s weights (an HF-format model directory).

A directory with no such file recommends nothing, exactly like a GGUF with no general.sampling.* keys: the absence of a recommendation is the normal case and must never be an error that stops a model from loading.

Trait Implementations

impl Clone for RecommendedSampling

fn clone(&self) -> RecommendedSampling

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for RecommendedSampling

impl Debug for RecommendedSampling

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for RecommendedSampling

fn default() -> RecommendedSampling

Returns the “default value” for a type.

impl PartialEq for RecommendedSampling

fn eq(&self, other: &RecommendedSampling) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for RecommendedSampling