Struct NvExt 

pub struct NvExt {

    pub ignore_eos: Option<bool>,
    pub top_k: Option<i32>,
    pub min_p: Option<f32>,
    pub repetition_penalty: Option<f32>,
    pub greed_sampling: Option<bool>,
    pub use_raw_prompt: Option<bool>,
    pub annotations: Option<Vec<String>>,
    pub backend_instance_id: Option<i64>,
    pub token_data: Option<Vec<u32>>,
    pub guided_json: Option<Value>,
    pub guided_regex: Option<String>,
    pub guided_grammar: Option<String>,
    pub guided_choice: Option<Vec<String>>,
    pub guided_decoding_backend: Option<String>,
    pub max_thinking_tokens: Option<u32>,
}

NVIDIA LLM extensions to the OpenAI API

Fields

ignore_eos: Option<bool>

If true, the model will ignore the end of string token and generate to max_tokens.

top_k: Option<i32>

min_p: Option<f32>

Relative probability floor

repetition_penalty: Option<f32>

How much to penalize tokens based on how frequently they occur in the text. A value of 1 means no penalty, while values larger than 1 discourage and values smaller encourage.

greed_sampling: Option<bool>

If true, sampling will be forced to be greedy. The backend is responsible for selecting the correct backend-specific options to implement this.

use_raw_prompt: Option<bool>

If true, the preproessor will try to bypass the prompt template and pass the prompt directly to to the tokenizer.

annotations: Option<Vec<String>>

Annotations User requests triggers which result in the request issue back out-of-band information in the SSE stream using the event: field.

backend_instance_id: Option<i64>

Targeted backend instance ID for the request If set, the request will be routed to backend instance with the given ID. If not set, the request will be routed to the best matching instance.

token_data: Option<Vec<u32>>

Pre-tokenized data to use instead of tokenizing the prompt If provided along with backend_instance_id, these tokens will be used directly and tokenization will be skipped.

guided_json: Option<Value>

Guided Decoding Options If specified, the output will be a JSON object. Can be a string, an object, or null.

guided_regex: Option<String>

If specified, the output will follow the regex pattern. Can be a string or null.

guided_grammar: Option<String>

If specified, the output will follow the context-free grammar. Can be a string or null.

guided_choice: Option<Vec<String>>

If specified, the output will be exactly one of the choices.

guided_decoding_backend: Option<String>

If specified, the backend to use for guided decoding, can be backends like xgrammar or custom guided decoding backend

max_thinking_tokens: Option<u32>

Maximum number of thinking tokens allowed NOTE: Currently passed through to backends as a no-op for future implementation

Implementations

impl NvExt

pub fn builder() -> NvExtBuilder

Trait Implementations

impl Clone for NvExt

fn clone(&self) -> NvExt

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NvExt

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for NvExt

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for NvExt

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for NvExt

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Validate for NvExt

fn validate(&self) -> Result<(), ValidationErrors>

impl<'v_a> ValidateArgs<'v_a> for NvExt

type Args = ()

fn validate_with_args(&self, args: Self::Args) -> Result<(), ValidationErrors>