Struct LLMEngineOutput 

pub struct LLMEngineOutput {
    pub token_ids: Vec<TokenIdType>,
    pub tokens: Option<Vec<TokenType>>,
    pub text: Option<String>,
    pub cum_log_probs: Option<f64>,
    pub log_probs: Option<LogProbs>,
    pub top_logprobs: Option<TopLogprobs>,
    pub finish_reason: Option<FinishReason>,
    pub index: Option<u32>,
}

The LLM engine and backnd with manage it’s own state, specifically translating how a given request/slot is managed on that particular backend.

For nvLLM’s purpose, it has a single tracable request_id as part of it’s context that has propaged through the service pipeline to the backend.

This is the minimal raw output from the LLM engine. The Backend may then apply multiple levels of post-processing before the BackendOutput is returns

Fields

token_ids: Vec<TokenIdType>

tokens: Option<Vec<TokenType>>

If the LLM Engine performs the detokenization, then this will have a Some of the detokenized text/tokens. If this value is None, then the Backend is responsible for detokenization.

text: Option<String>

cum_log_probs: Option<f64>

cumulative log probabilities

log_probs: Option<LogProbs>

Optional log probabilities

top_logprobs: Option<TopLogprobs>

finish_reason: Option<FinishReason>

index: Option<u32>

Implementations

impl LLMEngineOutput

pub fn cancelled() -> Self

pub fn stop() -> Self

pub fn length() -> Self

pub fn error(err_msg: String) -> Self

Trait Implementations

impl Clone for LLMEngineOutput

fn clone(&self) -> LLMEngineOutput

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LLMEngineOutput

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for LLMEngineOutput

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl MaybeError for LLMEngineOutput

fn from_err(err: Box<dyn Error + Send + Sync>) -> Self

Construct an instance from an error.

fn err(&self) -> Option<Error>

Construct into an error instance.

fn is_ok(&self) -> bool

Check if the current instance represents a success.

fn is_err(&self) -> bool

Check if the current instance represents an error.

impl PartialEq for LLMEngineOutput

fn eq(&self, other: &LLMEngineOutput) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for LLMEngineOutput

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for LLMEngineOutput