Struct Server 

pub struct Server { /* private fields */ }

Serve your own models from Rust, with everything the furnace binary does: dynamic batching, the Open Inference Protocol over HTTP (and gRPC), metrics, readiness, reloads, API keys, rate limits and TLS, all configured through ServerConfig.

// A Burn model's forward function, through furnace-burn
let model = furnace_burn::from_fn("digits", BackendKind::Cpu, &[1, 28, 28], &[10],
    move |x: Tensor<4>| net.forward(x))?;
Server::new().port(8080).model(model).serve().await?;

Server::router gives the HTTP API as an axum Router instead, to mount in your own application.

Implementations

impl Server

pub fn new() -> Self

A server with ServerConfig::default: 127.0.0.1:3000, batching off (see max_batch_size)

pub fn with_config(config: ServerConfig) -> Self

pub fn host(self, host: &str) -> Self

pub fn port(self, port: u16) -> Self

pub fn max_batch_size(self, items: usize) -> Self

The most items one batch carries, for the dynamic batching of concurrent requests (BatchConfig::max_batch_size); 1 turns batching off

pub fn request_timeout(self, timeout: Duration) -> Self

How long one inference may take before its request gets 504 Gateway Timeout (ServerConfig::request_timeout; none by default)

pub fn configure(self, change: impl FnOnce(&mut ServerConfig)) -> Self

Change any other setting

pub fn model(self, model: Model) -> Self

Serve model; the first model added is the one /predict uses

pub fn llm(self, engine: impl LlmEngine) -> Self

Serve an LLM (feature llm) on POST /v1/chat/completions, POST /v1/completions and GET /v1/models, next to the other models: a Scheduled step model, or any LlmEngine, or an Arc of one shared with a nerve Llm. A request without model goes to the first LLM added.

pub fn nerve(self, fabric: FabricBuilder) -> Self

Add neurons of your own to the server’s nervous system; a fabric that cannot be built makes serve and router return FurnaceError::Nerve. The server wires its models into the same fabric (an entrance in:{model}, a neuron model:{model} and the serve:respond effector), so your pathways and its requests share one system. The HTTP receptors become routes, the receptors start with the server, GET /nerve describes every neuron, pathway and measured latency, GET /nerve/effectors/{name}/events streams an SSE effector’s commands, and /metrics carries it all. A server may serve your neurons alone.

pub fn reloadable_model( self, model: Model, reload: impl Fn(Option<&Path>) -> Result<Model> + Send + Sync + 'static, ) -> Self

Serve model and let POST /admin/models/{name}/reload replace it with what reload returns (given the request’s weights path, if any). Needs an admin token in the config.

pub fn router(self) -> Result<Router>

The HTTP API as an axum router, to serve or mount yourself, or an error for a server with nothing to serve, as serve gives.

A rate limit without API keys tells clients apart by their address, which axum knows only when the router is served with ClientAddr as connect info; otherwise every client shares one budget:

let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await?;
axum::serve(
    listener,
    app.into_make_service_with_connect_info::<furnace::api::ClientAddr>(),
)
.await

pub async fn serve(self) -> Result<()>

Serve until Ctrl-C / SIGTERM

Trait Implementations

impl Default for Server

fn default() -> Self

Returns the “default value” for a type.