Struct PantryAPI

pub struct PantryAPI {
    pub client: Client<HttpConnector>,
    pub base_url: Option<String>,
}

PantryAPI is a thin wrapper, just meant to minimize retyping of client and baseurl in function calls. Feel free to make multiple, or to clone.

Fields

client: Client<HttpConnector>

base_url: Option<String>

Implementations

impl PantryAPI

pub fn new(base_url: Option<String>) -> Self

pub async fn register_user( &self, user_name: String, ) -> Result<UserInfo, PantryError>

Accessing the API requires a registered user demarcated by a user_id and an api_key.

This function supplies both. When using the API manually, you’ll probably also need to call PantryAPI::request_permissions to do anything useful.

Arguments

• user_name — used for debug output and manager display.

pub async fn request_permissions( &self, user_id: Uuid, api_key: String, requested_permissions: UserPermissions, ) -> Result<UserRequestStatus, PantryError>

Requests permissions. See the UserPermissions struct for more details. The system owner must accept the request (currently in the UI).

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• requested_permissions — The permissions this api user wants.

pub async fn request_download( &self, user_id: Uuid, api_key: String, llm_registry_entry: LLMRegistryEntry, ) -> Result<UserRequestStatus, PantryError>

Creates a request to download a new model. Must be accepted by the system owner (currently via the UI).

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_registry_entry — A valid LLM registry entry to download. This specifies the location of the model as well as any metadata. For better usability, try being comprehensive about this.

pub async fn request_load_flex( &self, user_id: Uuid, api_key: String, filter: Option<LLMFilter>, preference: Option<LLMPreference>, ) -> Result<UserRequestStatus, PantryError>

Requests a load, but doesn’t predetermine the exact LLM ahead of time.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• filter — An LLMFilter specifying hard requirements for the LLM.
• preference — An LLMPreference specifying soft requirements for the LLM.

pub async fn request_load( &self, user_id: Uuid, api_key: String, llm_id: Uuid, ) -> Result<UserRequestStatus, PantryError>

Requests Pantry to load a specific LLM.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_id — A UUID for the LLM you want to load. Find one via PantryAPI::get_available_llms.

pub async fn request_unload( &self, user_id: Uuid, api_key: String, llm_id: Uuid, ) -> Result<UserRequestStatus, PantryError>

Requests an LLM be shutdown, conserving resources. This should generally be non-destructive—Pantry attempts to save sessions to disc on shutdown.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_id — UUID of the LLM. Find running llms via PantryAPI::get_running_llms.

pub async fn get_request_status( &self, user_id: Uuid, api_key: String, request_id: Uuid, ) -> Result<UserRequestStatus, PantryError>

pub async fn get_llm_status( &self, user_id: Uuid, api_key: String, llm_id: Uuid, ) -> Result<LLMStatus, PantryError>

Gets the current status of an LLM

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user

pub async fn get_running_llms( &self, user_id: Uuid, api_key: String, ) -> Result<Vec<LLMStatus>, PantryError>

Gets currently running LLMs.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user

pub async fn get_available_llms( &self, user_id: Uuid, api_key: String, ) -> Result<Vec<LLMStatus>, PantryError>

Gets currently downloaded LLMs.

In order to create a session, these must first be activated, requiring the correct permissions. If you’re the system owner, you can also activate them manually using the UI.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user

pub async fn interrupt_session( &self, user_id: Uuid, api_key: String, llm_id: Uuid, session_id: Uuid, ) -> Result<LLMRunningStatus, PantryError>

Interrupts an ongoing inference session.

Internally this uses a cancellation callback to cancel inference after the next token. Depending on your system, this might take a moment, especially given that some tokens might already be inferred but not yet transmitted.

You must own the session to interrupt it.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_id — A UUID of an LLM. You should have gotten it from creating your session.
• session_id — A UUID of a session. You should have gotten it from creating your session.

pub async fn load_llm_flex( &self, user_id: Uuid, api_key: String, filter: Option<LLMFilter>, preference: Option<LLMPreference>, ) -> Result<LLMRunningStatus, PantryError>

Loads an LLM.

Requires the UserPermissions::perm_load_llm permission.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• filter — A LLMFilter object, for what must be true of an LLM to load it.
• preference — A LLMPreference object, for how to rank and then select from the LLMs that pass the filter.

pub async fn load_llm( &self, user_id: Uuid, api_key: String, llm_id: String, ) -> Result<LLMRunningStatus, PantryError>

Loads an LLM.

Requires UserPermissions::perm_load_llm.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_id — UUID or ID of an LLM. Will fail for duplicate IDs.

pub async fn unload_llm( &self, user_id: Uuid, api_key: String, llm_id: String, ) -> Result<LLMStatus, PantryError>

Unloads an LLM, conserving resources.

Requires UserPermissions::perm_unload_llm.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_id — UUID or model id of an LLM.

pub async fn download_llm( &self, user_id: Uuid, api_key: String, llm_registry_entry: LLMRegistryEntry, ) -> Result<Value, PantryError>

Downloads an LLM.

Requires UserPermissions::perm_download_llm.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_registry_entry — LLMRegistryEntry for the LLM. The only “mandatory” fields are the LLMRegistryEntry::connector_type and LLMRegistryEntry::id. If the connector type is crate::interface::LLMConnectorType::LLMrs, config must include the key model_architecture. For more details see the rustformers/llm documentation

pub async fn create_session( &self, user_id: Uuid, api_key: String, user_session_parameters: HashMap<String, Value>, ) -> Result<CreateSessionResponse, PantryError>

Creates a session, using the best currently running LLM.

Requires UserPermissions::perm_session.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• user_session_parameters — A hashmap of requested parameters. The returning LLMStatus object will inform which ones got accepted by the LLM.

pub async fn create_session_id( &self, user_id: Uuid, api_key: String, llm_id: Uuid, user_session_parameters: HashMap<String, Value>, ) -> Result<CreateSessionResponse, PantryError>

Creates a session, using the LLM with the given id. If the LLM doesn’t exist or isn’t running, errors.

Requires UserPermissions::perm_session.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user OR a machine-id. The API will attempt to match on UUID first, otherwise treat it as a flex request matching on id.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_id — A UUID for which LLM to use.
• user_session_parameters — A hashmap of requested parameters. The returning LLMStatus object will inform which ones got accepted by the LLM.

pub async fn create_session_flex( &self, user_id: Uuid, api_key: String, filter: Option<LLMFilter>, preference: Option<LLMPreference>, user_session_parameters: HashMap<String, Value>, ) -> Result<CreateSessionResponse, PantryError>

Creates a session based on filter and preference. Selects only from currently running LLMs.

Requires UserPermissions::perm_session.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• filter — A LLMFilter object, for what must be true of an LLM to use it.
• preference — A LLMPreference object, for how to rank and then select from the LLMs
• user_session_parameters — A hashmap of requested parameters. The returning LLMStatus object will inform which ones got accepted by the LLM.

pub async fn prompt_session_stream( &self, user_id: Uuid, api_key: String, session_id: Uuid, llm_uuid: String, prompt: String, parameters: HashMap<String, Value>, ) -> Result<LLMEventStream, PantryError>

Prompts a session, triggering inference by the LLM.

Requires UserPermissions::perm_session.

Session must be running first, call PantryAPI::create_session, PantryAPI::create_session_id, or PantryAPI::create_session_flex first.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• session_id — A UUID representing the session. Obtained by calling PantryAPI::create_session or its variants.
• llm_uuid — UUID of the llm. Must match the call used to make the session.
• prompt — Prompt for the LLM. Pantry does no preprompting, so if you want a chatbot style response, you’ll need to insert a chatbot type prompt then whatever the user requested.
• parameters — Things like temperature or k value. Whats available varies by LLM, you can find out what an LLM has either in the UI or in the user_parameters and user_session_parameters vectors of an LLMStatus.

pub async fn bare_model( &self, user_id: Uuid, api_key: String, llm_id: String, ) -> Result<BareModelResponse, PantryError>

Acquire a bare model.

In practice this means a file path to a GGML file that you can then run yourself with whatever your preferred LLM runner is.

Requires UserPermissions::perm_bare_model.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• llm_id — UUID of an LLM.

pub async fn bare_model_flex( &self, user_id: Uuid, api_key: String, filter: Option<LLMFilter>, preference: Option<LLMPreference>, ) -> Result<BareModelResponse, PantryError>

Returns a bare model based on filter and preference.

Requires UserPermissions::perm_bare_model.

Arguments

• user_id — A UUID, obtained from PantryAPI::register_user.
• api_key — An API key, obtained from PantryAPI::register_user
• filter — A LLMFilter object, for what must be true of an LLM to use it.
• preference — A LLMPreference object, for how to rank and then select from the LLMs

pub async fn get_or_download_llm( &self, user_id: Uuid, api_key: String, llm_registry_entry: LLMRegistryEntry, ) -> Result<Value, PantryError>

Trait Implementations

impl Clone for PantryAPI

fn clone(&self) -> PantryAPI

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PantryAPI

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.