pub struct ChatCompletionRequest {Show 14 fields
pub model: String,
pub messages: Vec<Message>,
pub stream: Option<bool>,
pub stream_options: Option<StreamOptions>,
pub temperature: Option<f32>,
pub top_p: Option<f32>,
pub top_k: Option<u32>,
pub seed: Option<u64>,
pub max_tokens: Option<u32>,
pub max_completion_tokens: Option<u32>,
pub stop: Option<StringOrArray>,
pub n: Option<u32>,
pub tools: Option<Vec<Value>>,
pub tool_choice: Option<Value>,
}Expand description
A POST /v1/chat/completions body.
Unknown fields are ignored rather than rejected — a client that sends a parameter this build does not implement gets an answer, not a 400 — so the fields listed here are the ones that change behaviour, not the ones that are legal.
Fields§
§model: StringWhatever the client called the model. Echoed back unexamined: this server serves exactly one install, so validating the name could only refuse requests it is able to answer.
messages: Vec<Message>The conversation. Must be non-empty.
stream: Option<bool>Stream the reply as SSE.
stream_options: Option<StreamOptions>Streaming extras.
temperature: Option<f32>Softmax temperature.
top_p: Option<f32>Nucleus-sampling mass.
top_k: Option<u32>Top-k cutoff. Not an OpenAI field; accepted because llama.cpp accepts it and local tooling sends it.
seed: Option<u64>PRNG seed.
max_tokens: Option<u32>Reply length cap, the older spelling.
max_completion_tokens: Option<u32>Reply length cap, the current spelling. Wins when both appear.
stop: Option<StringOrArray>Stop sequences.
n: Option<u32>How many completions to return. Anything above 1 is refused.
tools: Option<Vec<Value>>Tool definitions, preserved verbatim.
Held as raw JSON rather than a typed schema because the renderer takes
them that way: each definition is serialized into the # Tools system
block exactly as it arrived, so a Rust shape here could only lose
fields (strict, vendor extensions) that the reference template keeps.
tool_choice: Option<Value>Tool selection policy. Only auto and none are honourable here; see
ChatCompletionRequest::check_tool_choice.
Implementations§
Source§impl ChatCompletionRequest
impl ChatCompletionRequest
Sourcepub fn from_json(body: &str) -> Result<Self, ServerError>
pub fn from_json(body: &str) -> Result<Self, ServerError>
Parse a request body.
Every byte here is untrusted, so serde’s failure is captured as a message rather than propagated as a panic or an opaque 500.
Sourcepub fn validate(&self) -> Result<(), ServerError>
pub fn validate(&self) -> Result<(), ServerError>
Check everything that can be checked without a tokenizer.
Called before any work is scheduled, so a request that cannot be served never reaches the decode loop and never occupies the single session.
Sourcepub fn check_tools(&self) -> Result<(), ServerError>
pub fn check_tools(&self) -> Result<(), ServerError>
Refuse a tool definition whose name the model could not call back.
Separate from validate so crate::prompt::build,
which is the thing that actually renders these, enforces it too rather
than trusting that validation ran.
Sourcepub fn check_tool_choice(&self) -> Result<(), ServerError>
pub fn check_tool_choice(&self) -> Result<(), ServerError>
Refuse a tool_choice this build cannot honour.
auto, none and absent are accepted and all render the same prompt,
because that is what the template does: it branches on tools alone
and has no way to say “you must call something”. required and a named
function therefore cannot be expressed at all, and serving them by
letting the model choose freely would drop the one constraint the
request was about.
Sourcepub fn include_usage(&self) -> bool
pub fn include_usage(&self) -> bool
Whether to emit the usage-only chunk. Absent means false.
Sourcepub fn max_new_tokens(&self) -> Option<u32>
pub fn max_new_tokens(&self) -> Option<u32>
The reply length cap.
max_completion_tokens is the current spelling and max_tokens the
deprecated alias; when both arrive the newer one wins, because a client
that sends both is an SDK filling in a compatibility field behind a
caller who set the modern one.
Sourcepub fn stop_sequences(&self) -> &[String]
pub fn stop_sequences(&self) -> &[String]
The stop sequences, empty when none were sent.
Read by validate only, to refuse a non-empty one.
When the runtime grows stop strings this becomes the accessor that
feeds them; until then it exists so the refusal has one definition of
“sent a stop sequence” across both stop spellings.
Trait Implementations§
Source§impl Clone for ChatCompletionRequest
impl Clone for ChatCompletionRequest
Source§impl Debug for ChatCompletionRequest
impl Debug for ChatCompletionRequest
Source§impl<'de> Deserialize<'de> for ChatCompletionRequest
impl<'de> Deserialize<'de> for ChatCompletionRequest
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Source§impl PartialEq for ChatCompletionRequest
impl PartialEq for ChatCompletionRequest
Source§impl Serialize for ChatCompletionRequest
impl Serialize for ChatCompletionRequest
impl StructuralPartialEq for ChatCompletionRequest
Auto Trait Implementations§
impl Freeze for ChatCompletionRequest
impl RefUnwindSafe for ChatCompletionRequest
impl Send for ChatCompletionRequest
impl Sync for ChatCompletionRequest
impl Unpin for ChatCompletionRequest
impl UnsafeUnpin for ChatCompletionRequest
impl UnwindSafe for ChatCompletionRequest
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> DeserializeOwned for Twhere
T: for<'de> Deserialize<'de>,
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more