Enum PromptType 

pub enum PromptType {
    TokenIds(Vec<TokenIdType>),
    Raw(String),
    Completion(CompletionContext),
    ChatCompletion(ChatContext),
    CustomJson(Value),
}

LLM Inference Engines can accept a variety of input types. Not all Engines will support all input types. For example, the trtllm::AsyncEngine only supports PromptType::Tokens as an input type. The higher-level Backend class is a general wrapper around Engines that will enable many of the input options that require pre/postprocessing.

Variants

TokenIds(Vec<TokenIdType>)

If allowed, this input type allowed the caller to pass a list of token_ids directly to the inference engine. This is an advanced feature that requires the caller to handle all of the necessary prompt formatting and tokenization.

Raw(String)

If allowed, the raw text will be tokenized and converted to token_ids without any additional preprocessing. This is an advanced features that requires the caller to correctly format the prompt as defined by the model.

Completion(CompletionContext)

If allowed, the CompletionContext will be preprocessed server-side. If the Model trait requires_prompt_template returns true then the CompletionContext will be used to to render the formatted prompt from the template. Completion is the preferred PromptType for single turn completions.

ChatCompletion(ChatContext)

If allowed, the ChatContext will be preprocessed server-side. Most chat models will have a predefined prompt format/structure. If the Model trait requires_prompt_template returns true then the ChatContext will be used to to render the formatted prompt from the template. ChatCompletion is the preferred PromptType for multi-turn completions.

CustomJson(Value)

If allowed, then Model::requires_prompt_template() must also return true. The serde_json::Value will be passed directly the prompt template. This allows for a complete generic data model and prompt template to be passed to be defined and used by the server.

Trait Implementations

impl Clone for PromptType

fn clone(&self) -> PromptType

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PromptType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for PromptType

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl From<CompletionContext> for PromptType

fn from(context: CompletionContext) -> Self

Converts to this type from the input type.

impl PartialEq for PromptType

fn eq(&self, other: &PromptType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for PromptType

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Eq for PromptType

impl StructuralPartialEq for PromptType