Struct LocalPrompt

pub struct LocalPrompt {
    pub generation_prefix: Mutex<Option<String>>,
    pub built_prompt_string: Mutex<Option<String>>,
    pub built_prompt_as_tokens: Mutex<Option<Vec<u32>>>,
    pub total_prompt_tokens: Mutex<Option<u64>>,
    /* private fields */
}

A prompt formatter for local LLMs that use chat templates.

LocalPrompt handles formatting messages according to a model’s chat template, managing special tokens (BOS, EOS, UNK), and supporting generation prefixes. Unlike API prompts, local prompts need to handle the specific formatting requirements and token conventions of locally-run models.

The struct maintains both string and tokenized representations of the built prompt, along with thread-safe interior mutability for managing prompt state. It supports token counting and generation prefix management for model outputs.

Fields

generation_prefix: Mutex<Option<String>>

built_prompt_string: Mutex<Option<String>>

built_prompt_as_tokens: Mutex<Option<Vec<u32>>>

total_prompt_tokens: Mutex<Option<u64>>

Implementations

impl LocalPrompt

pub fn get_built_prompt(&self) -> Result<String, Error>

Retrieves the built prompt as a formatted string.

Returns the complete prompt string with all messages formatted according to the chat template, including any special tokens and generation prefix.

Returns

Returns Ok(String) containing the formatted prompt string.

Errors

Returns an error if the prompt has not been built yet.

pub fn get_built_prompt_as_tokens(&self) -> Result<Vec<u32>, Error>

Retrieves the built prompt as a vector of tokens.

Returns the complete prompt converted to model tokens using the configured tokenizer. This is useful for operations that need to work directly with token IDs rather than text.

Returns

Returns Ok(Vec<u32>) containing the token IDs for the prompt.

Errors

Returns an error if the prompt has not been built yet.

pub fn get_total_prompt_tokens(&self) -> Result<u64, Error>

Gets the total number of tokens in the built prompt.

Returns the exact token count of the built prompt, which is useful for ensuring prompts stay within model context limits. This count reflects all content, special tokens, and any generation prefix.

Returns

Returns Ok(u64) containing the total token count.

Errors

Returns an error if the prompt has not been built yet.

Trait Implementations

impl Clone for LocalPrompt

fn clone(&self) -> LocalPrompt

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Display for LocalPrompt

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Serialize for LocalPrompt

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.