Crate llm_prompt

Structs

• ApiPrompt
  A prompt formatter for API-based language models that follow OpenAI’s message format.
• LlmPrompt
  A prompt management system that supports both API-based LLMs (like OpenAI) and local LLMs.
• LocalPrompt
  A prompt formatter for local LLMs that use chat templates.
• MaxTokenState
• PromptMessage
  An individual message within a prompt sequence.
• PromptMessages
  A collection of prompt messages with thread-safe mutability.

Enums

• PromptMessageType
  Represents the type of message in a prompt sequence.
• RequestTokenLimitError
• TextConcatenator
  Controls how text segments are joined together in prompt messages.

Traits

• PromptTokenizer
  A trait for tokenizers that can be used with the prompt management system.
• TextConcatenatorTrait
  Provides methods for managing text concatenation behavior.

Functions

• apply_chat_template
  Applies a chat template to a message, given a message and a chat template.
• check_and_get_max_tokens
  Sets and validates the ‘max_tokens’ or ‘n_ctx’ or ‘n_predict’ parameter for a request. First, it checks that the total_prompt_tokens is less than the ctx_size - safety_tokens. Then returns ‘available_tokens’ as the lower of either: ctx_size - total_prompt_tokens - safety_tokens or if it’s provided, inference_ctx_size. If ‘requested_tokens’ is provided, ‘requested_tokens’ is returned if less than ‘available_tokens’. If ‘requested_tokens’ is ‘None’ or ‘requested_tokens’ is greater than ‘available_tokens’, ‘available_tokens’ is returned.