Crate inference_gateway_sdk

Expand description

Inference Gateway SDK for Rust

This crate provides a Rust client for the Inference Gateway API, allowing interaction with various LLM providers through a unified interface.

Data types in [crate::generated::schemas] are generated from the upstream openapi.yaml and re-exported at the crate root. Run task generate-types to regenerate them after a spec bump.

Modules§

defaults: Generation of default values for serde.
error: Error types.

Structs§

CacheControl: Cache control settings for prompt caching. Currently only ephemeral caching is supported.
ChatCompletionChoice: ChatCompletionChoice
ChatCompletionChoiceLogprobs: Log probability information for the choice.
ChatCompletionMessageToolCall: ChatCompletionMessageToolCall
ChatCompletionMessageToolCallChunk: ChatCompletionMessageToolCallChunk
ChatCompletionMessageToolCallFunction: The function that the model called.
ChatCompletionNamedToolChoice: Specifies a tool the model should use. Use to force the model to call a specific function.
ChatCompletionNamedToolChoiceFunction: ChatCompletionNamedToolChoiceFunction
ChatCompletionStreamChoice: ChatCompletionStreamChoice
ChatCompletionStreamChoiceLogprobs: Log probability information for the choice.
ChatCompletionStreamOptions: Options for streaming response. Only set this when you set stream: true.
ChatCompletionStreamResponseDelta: A chat completion delta generated by streamed model responses.
ChatCompletionTokenLogprob: ChatCompletionTokenLogprob
ChatCompletionTokenLogprobTopLogprobsItem: ChatCompletionTokenLogprobTopLogprobsItem
ChatCompletionTool: ChatCompletionTool
CompletionUsage: Usage statistics for the completion request.
CompletionUsageCompletionTokensDetails: Breakdown of tokens used in a completion.
CompletionUsagePromptTokensDetails: Breakdown of tokens used in the prompt.
Config: Config
ContextWindow: Context window information for a model
CreateChatCompletionRequest: CreateChatCompletionRequest
CreateChatCompletionResponse: Represents a chat completion response returned by model, based on the provided input.
CreateChatCompletionStreamResponse: Represents a streamed chunk of a chat completion response returned by the model, based on the provided input.
CreateImageEditRequest: Request for InferenceGatewayAPI::create_image_edit.
CreateImageRequest: Request body for creating an image via the OpenAI-compatible Images API.
CreateImageVariationRequest: Request for InferenceGatewayAPI::create_image_variation.
CreateMessagesRequest: Request body for creating a message via the Anthropic-compatible Messages API.
CreateMessagesRequestThinking: Configuration for extended thinking.
CreateResponseRequest: Request body for creating a model response via the Responses API.
CreateSpeechRequest: Request body for generating speech audio via the OpenAI-compatible Audio API.
CreateSpeechRequestInput: The text to synthesize into audio (4096 characters maximum).
CreateSpeechRequestInstructions: Control the voice of your generated audio with additional instructions. Does not work with tts-1 or tts-1-hd.
CreateSpeechRequestLanguage: ISO 639-1 code for the language of the generated speech. Non-standard extension: OpenAI’s speech API has no language field (the name matches its transcription API). Forwarded to the provider as-is; the gateway’s built-in local engine (local/qwen3-tts) supports zh, en, de, it, pt, es, ja, ko, fr and ru, and rejects other codes.
Endpoints: Endpoints
Error: Error
FunctionObject: FunctionObject
FunctionParameters: The parameters the functions accepts, described as a JSON Schema object. See the guide for examples, and the JSON Schema reference for documentation about the format. Omitting parameters defines a function with an empty parameter list.
Image: Represents the url or the content of an image generated by the Images API.
ImageContentPart: Image content part
ImageUrl: Image URL configuration
ImagesResponse: Represents the result of an image generation request.
ImagesResponseUsage: Usage statistics for the image generation request.
ImagesResponseUsageInputTokensDetails: A detailed breakdown of the input tokens.
InferenceGatewayClient: Client for interacting with the Inference Gateway API
ListModelsResponse: Response structure for listing models
ListToolsResponse: Response structure for listing MCP tools
McpTool: An MCP tool definition
Message: Message structure for provider requests
MessagesDocumentBlock: A document content block in a Messages API request.
MessagesDocumentSource: The source of a document content block. Can be a base64-encoded document or a URL.
MessagesError: An error response in the Anthropic error format.
MessagesErrorError: The error details.
MessagesImageBlock: An image content block in a Messages API request.
MessagesImageSource: The source of an image content block. Can be a base64-encoded image or a URL.
MessagesMessage: A message in a Messages API request.
MessagesMetadata: Metadata for a Messages API request.
MessagesOutputConfig: Output configuration for a Messages API request.
MessagesRedactedThinkingBlock: A redacted thinking content block in a Messages API request or response. Emitted when thinking content is encrypted for safety reasons; must be passed back unchanged in multi-turn conversations.
MessagesResponse: A message response from the Anthropic-compatible Messages API.
MessagesStreamEvent: A server-sent event emitted while streaming a Messages API response. The Anthropic Messages API emits a sequence of typed events (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop, ping).
MessagesStreamEventDelta: Present in content_block_delta and message_delta events. Contains the incremental update.
MessagesTextBlock: A text content block in a Messages API request or response.
MessagesThinkingBlock: A thinking content block in a Messages API request or response.
MessagesTool: A tool definition in the Messages API format. Uses the same function tool shape as the Responses API but with an optional cache_control field for prompt caching.
MessagesToolResultBlock: A tool result content block in a Messages API request.
MessagesToolUseBlock: A tool use content block in a Messages API request or response.
MessagesUsage: Token usage statistics for a Messages API response, including cache metrics.
Model: Common model information
ModelModalities: The input and output modalities of a model, mirroring the models.dev dataset shape. Vision models accept image in input; image-generation models list image in output - when output carries image but not text, the model only generates images and cannot chat.
Pricing: Pricing information for a model
ProviderSpecificResponse: Provider-specific response format. Examples:
Response: Represents a model response returned by the Responses API.
ResponseError: An error object returned when the model fails to generate a response.
ResponseFormatJsonObject: JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.
ResponseFormatJsonSchema: JSON Schema response format. Used to generate structured JSON responses.
ResponseFormatJsonSchemaJsonSchema: Structured Outputs configuration options, including a JSON Schema.
ResponseFormatJsonSchemaSchema: The schema for the response format, described as a JSON Schema object.
ResponseFormatText: Default response format. Used to generate text responses.
ResponseFunctionToolCall: A tool call to a function generated by the model.
ResponseIncompleteDetails: Details about why the response is incomplete.
ResponseInputImage: An image input to the model.
ResponseInputItem: A single input item. Most commonly an input message with a role and content.
ResponseInputText: A text input to the model.
ResponseOutputMessage: An output message from the model.
ResponseOutputRefusal: A refusal generated by the model.
ResponseOutputText: A text output from the model.
ResponseReasoning: Configuration options for reasoning models.
ResponseReasoningItem: A reasoning item describing the model’s chain of thought.
ResponseReasoningSummaryPart: A summary part of a reasoning item.
ResponseStreamEvent: A server-sent event emitted while streaming a response. The Responses API emits a sequence of typed events (for example response.created, response.output_text.delta, and response.completed). This schema models the common event envelope; which fields are populated depends on the event type.
ResponseTextConfig: Configuration options for a text response from the model. Can be plain text or structured JSON data.
ResponseTextConfigFormat: An object specifying the format that the model must output.
ResponseTool: A tool the model may call. Only function tools are modeled here. Note the Responses API uses a flattened function tool shape (name, description, and parameters at the top level) rather than nesting them under a function object as /chat/completions does.
ResponseUsage: Token usage details for the response.
ResponseUsageInputTokensDetails: A detailed breakdown of the input tokens.
ResponseUsageOutputTokensDetails: A detailed breakdown of the output tokens.
SSEvents: Stream of Server-Sent Events (SSE) yielded by InferenceGatewayAPI::generate_content_stream.
SsEvent: SsEvent
TextContentPart: Text content part
ToolCallExtraContent: Provider-specific opaque data attached to a tool call. The contents are not interpreted by the gateway, but must be echoed back verbatim on the next request that references this tool call. Currently used by Google Gemini extended-thinking models to carry the per-call thought_signature. Other providers may ignore the field.
ToolCallExtraContentGoogle: Google Gemini-specific extra content.

Enums§

CacheControlType: The cache control type. Currently only ephemeral.
ChatCompletionToolChoiceOption: Controls which (if any) tool is called by the model. none means the model will not call any tool and instead generates a message. auto means the model can pick between generating a message or calling one or more tools. required means the model must call one or more tools. Specifying a particular tool via {"type": "function", "function": {"name": "my_function"}} forces the model to call that tool. none is the default when no tools are present. auto is the default if tools are present.
ChatCompletionToolChoiceOptionString: none means the model will not call any tool and instead generates a message. auto means the model can pick between generating a message or calling one or more tools. required means the model must call one or more tools.
ChatCompletionToolType: The type of the tool. Currently, only function is supported.
ContentPart: A content part within a multimodal message
ContextWindowSource: Source of the context window information
CreateChatCompletionRequestReasoningEffort: Constrains effort on reasoning for reasoning models. Currently supported values are minimal, low, medium, and high. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.
CreateChatCompletionRequestResponseFormat: An object specifying the format that the model must output. Setting to { "type": "json_schema", "json_schema": {...} } enables Structured Outputs which guarantees the model will match your supplied JSON schema. Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON.
CreateChatCompletionRequestStop: Up to 4 sequences where the API will stop generating further tokens.
CreateImageRequestQuality: The quality of the image. auto selects the best quality for the model. The GPT image models support low, medium, and high; dall-e-3 supports standard and hd; dall-e-2 supports only standard.
CreateImageRequestResponseFormat: The format in which the generated images are returned. Must be one of url or b64_json.
CreateMessagesRequestSystem: The system prompt. Can be a string or an array of system content blocks (for prompt caching).
CreateMessagesRequestThinkingType: Always enabled.
CreateSpeechRequestResponseFormat: The audio format of the response.
FinishReason: The reason the model stopped generating tokens. This will be stop if the model hit a natural stop point or a provided stop sequence, length if the maximum number of tokens specified in the request was reached, content_filter if content was omitted due to a flag from our content filters, tool_calls if the model called a tool.
GatewayError: Custom error types for the Inference Gateway SDK
ImageContentPartType: Content type identifier
ImageSize: The size of the generated images. The GPT image models support 1024x1024, 1536x1024, 1024x1536, and auto; gpt-image-2 also accepts arbitrary WIDTHxHEIGHT values such as 1536x864. dall-e-2 supports 256x256, 512x512, and 1024x1024; dall-e-3 supports 1024x1024, 1792x1024, and 1024x1792.
ImageUrlDetail: Image detail level for vision processing
MessageContent: Message content - either text or multimodal content parts
MessageRole: Role of the message sender
MessagesDocumentBlockType: Content type identifier. Always document.
MessagesDocumentSourceType: The source type.
MessagesErrorType: Always error.
MessagesImageBlockType: Content type identifier. Always image.
MessagesImageSourceType: The source type.
MessagesMessageContent: The content of the message. Can be a string or an array of content blocks.
MessagesMessageRole: The role of the message sender.
MessagesOutputConfigEffort: Constrains how much effort the model spends on reasoning. Lower effort yields faster responses and fewer reasoning tokens.
MessagesRedactedThinkingBlockType: Content type identifier. Always redacted_thinking.
MessagesRequestContentBlock: A content block within a Messages API request message.
MessagesResponseContentBlock: A content block within a Messages API response.
MessagesResponseRole: Always assistant.
MessagesResponseStopReason: The reason the model stopped generating.
MessagesResponseType: Always message.
MessagesStreamEventType: The type of the streamed event.
MessagesTextBlockType: Content type identifier. Always text.
MessagesThinkingBlockType: Content type identifier. Always thinking.
MessagesToolChoice: Controls which (if any) tool is called by the model. auto means the model can decide, any means the model must use a tool, and tool forces a specific tool.
MessagesToolChoiceObjectType: Always tool.
MessagesToolChoiceString: The tool choice mode.
MessagesToolResultBlockContent: The result content. Can be a string or an array of content blocks.
MessagesToolResultBlockType: Content type identifier. Always tool_result.
MessagesToolUseBlockType: Content type identifier. Always tool_use.
Modality: A single input or output modality
PricingSource: Source of the pricing information
Provider: Provider
ProviderAuthType: Authentication type for providers
ResponseFormatJsonObjectType: The type of response format being defined. Always json_object.
ResponseFormatJsonSchemaType: The type of response format being defined. Always json_schema.
ResponseFormatTextType: The type of response format being defined. Always text.
ResponseFunctionToolCallStatus: The status of the function tool call.
ResponseFunctionToolCallType: The type of the output item. Always function_call.
ResponseInput: Text, image, or file inputs to the model. Either a single text prompt or a list of input items representing a (possibly batched) conversation.
ResponseInputContentPart: A content part within an input message.
ResponseInputImageDetail: The detail level of the image to send to the model.
ResponseInputImageType: The type of the input item. Always input_image.
ResponseInputMessageContent: Text or multimodal content for an input message. Either a string or a list of content parts.
ResponseInputTextType: The type of the input item. Always input_text.
ResponseOutputContent: A content part of an output message.
ResponseOutputItem: An output item generated by the model: an output message, a function tool call, or a reasoning item.
ResponseOutputMessageRole: The role of the output message. Always assistant.
ResponseOutputMessageStatus: The status of the message.
ResponseOutputMessageType: The type of the output item. Always message.
ResponseOutputRefusalType: The type of the refusal. Always refusal.
ResponseOutputTextType: The type of the output text. Always output_text.
ResponseReasoningEffort: Constrains the effort on reasoning for reasoning models. Reducing effort can result in faster responses and fewer reasoning tokens.
ResponseReasoningItemStatus: The status of the reasoning item.
ResponseReasoningItemType: The type of the output item. Always reasoning.
ResponseReasoningSummary: A summary of the reasoning performed by the model, useful for debugging and understanding the model’s reasoning process.
ResponseReasoningSummaryPartType: The type of the summary. Always summary_text.
ResponseRole: The role of the message input.
ResponseStatus: The status of the response generation.
ResponseTextConfigFormatType: The type of response format being defined.
ResponseToolChoice: How the model should select which tool (or tools) to use. Either a mode string (none, auto, required) or an object forcing a specific tool.
ResponseToolChoiceObjectType: ResponseToolChoiceObjectType
ResponseToolChoiceString: The tool-choice mode.
ResponseToolType: The type of the tool. Currently only function.
SsEventEvent: SsEventEvent
TextContentPartType: Content type identifier

Traits§

InferenceGatewayAPI: Core API interface for the Inference Gateway

Crate inference_gateway_sdk

Crate inference_gateway_sdk Copy item path

Modules§

Structs§

Enums§

Traits§

Crate inference_gateway_sdk