Crate inference_gateway_sdk 

Inference Gateway SDK for Rust

This crate provides a Rust client for the Inference Gateway API, allowing interaction with various LLM providers through a unified interface.

Data types in [crate::generated::schemas] are generated from the upstream openapi.yaml and re-exported at the crate root. Run task generate-types to regenerate them after a spec bump.

Modules

defaults

Generation of default values for serde.

error

Error types.

Structs

ChatCompletionChoice

ChatCompletionChoice

ChatCompletionChoiceLogprobs

Log probability information for the choice.

ChatCompletionMessageToolCall

ChatCompletionMessageToolCall

ChatCompletionMessageToolCallChunk

ChatCompletionMessageToolCallChunk

ChatCompletionMessageToolCallFunction

The function that the model called.

ChatCompletionNamedToolChoice

Specifies a tool the model should use. Use to force the model to call a specific function.

ChatCompletionNamedToolChoiceFunction

ChatCompletionNamedToolChoiceFunction

ChatCompletionStreamChoice

ChatCompletionStreamChoice

ChatCompletionStreamChoiceLogprobs

Log probability information for the choice.

ChatCompletionStreamOptions

Options for streaming response. Only set this when you set stream: true.

ChatCompletionStreamResponseDelta

A chat completion delta generated by streamed model responses.

ChatCompletionTokenLogprob

ChatCompletionTokenLogprob

ChatCompletionTokenLogprobTopLogprobsItem

ChatCompletionTokenLogprobTopLogprobsItem

ChatCompletionTool

ChatCompletionTool

CompletionUsage

Usage statistics for the completion request.

Config

Config

CreateChatCompletionRequest

CreateChatCompletionRequest

CreateChatCompletionResponse

Represents a chat completion response returned by model, based on the provided input.

CreateChatCompletionStreamResponse

Represents a streamed chunk of a chat completion response returned by the model, based on the provided input.

Endpoints

Endpoints

Error

Error

FunctionObject

FunctionObject

FunctionParameters

The parameters the functions accepts, described as a JSON Schema object. See the guide for examples, and the JSON Schema reference for documentation about the format. Omitting parameters defines a function with an empty parameter list.

ImageContentPart

Image content part

ImageUrl

Image URL configuration

InferenceGatewayClient

Client for interacting with the Inference Gateway API

ListModelsResponse

Response structure for listing models

ListToolsResponse

Response structure for listing MCP tools

McpTool

An MCP tool definition

Message

Message structure for provider requests

Model

Common model information

ProviderSpecificResponse

Provider-specific response format. Examples:

ResponseFormatJsonObject

JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

ResponseFormatJsonSchema

JSON Schema response format. Used to generate structured JSON responses.

ResponseFormatJsonSchemaJsonSchema

Structured Outputs configuration options, including a JSON Schema.

ResponseFormatJsonSchemaSchema

The schema for the response format, described as a JSON Schema object.

ResponseFormatText

Default response format. Used to generate text responses.

SSEvents

Stream of Server-Sent Events (SSE) yielded by InferenceGatewayAPI::generate_content_stream.

SsEvent

SsEvent

TextContentPart

Text content part

ToolCallExtraContent

Provider-specific opaque data attached to a tool call. The contents are not interpreted by the gateway, but must be echoed back verbatim on the next request that references this tool call. Currently used by Google Gemini extended-thinking models to carry the per-call thought_signature. Other providers may ignore the field.

ToolCallExtraContentGoogle

Google Gemini-specific extra content.

Enums

ChatCompletionToolChoiceOption

Controls which (if any) tool is called by the model. none means the model will not call any tool and instead generates a message. auto means the model can pick between generating a message or calling one or more tools. required means the model must call one or more tools. Specifying a particular tool via {"type": "function", "function": {"name": "my_function"}} forces the model to call that tool. none is the default when no tools are present. auto is the default if tools are present.

ChatCompletionToolChoiceOptionString

none means the model will not call any tool and instead generates a message. auto means the model can pick between generating a message or calling one or more tools. required means the model must call one or more tools.

ChatCompletionToolType

The type of the tool. Currently, only function is supported.

ContentPart

A content part within a multimodal message

CreateChatCompletionRequestReasoningEffort

Constrains effort on reasoning for reasoning models. Currently supported values are minimal, low, medium, and high. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

CreateChatCompletionRequestResponseFormat

An object specifying the format that the model must output. Setting to { "type": "json_schema", "json_schema": {...} } enables Structured Outputs which guarantees the model will match your supplied JSON schema. Setting to { "type": "json_object" } enables the older JSON mode, which ensures the message the model generates is valid JSON.

CreateChatCompletionRequestStop

Up to 4 sequences where the API will stop generating further tokens.

FinishReason

The reason the model stopped generating tokens. This will be stop if the model hit a natural stop point or a provided stop sequence, length if the maximum number of tokens specified in the request was reached, content_filter if content was omitted due to a flag from our content filters, tool_calls if the model called a tool.

GatewayError

Custom error types for the Inference Gateway SDK

ImageContentPartType

Content type identifier

ImageUrlDetail

Image detail level for vision processing

MessageContent

Message content - either text or multimodal content parts

MessageRole

Role of the message sender

Provider

Provider

ProviderAuthType

Authentication type for providers

ResponseFormatJsonObjectType

The type of response format being defined. Always json_object.

ResponseFormatJsonSchemaType

The type of response format being defined. Always json_schema.

ResponseFormatTextType

The type of response format being defined. Always text.

SsEventEvent

SsEventEvent

TextContentPartType

Content type identifier

Traits

InferenceGatewayAPI

Core API interface for the Inference Gateway