Skip to main content

Crate hearth_api

Crate hearth_api 

Source
Expand description

One OpenAI-compatible port in front of a hearth fleet.

The proxying is the boring part. What makes this worth having is that when hearth cannot serve a request, the HTTP response says which of four very different things happened — still loading, evicted, GPU reclaimed, or will never fit — and whether retrying here could ever work. See gateway.

Re-exports§

pub use gateway::decide;
pub use gateway::Decision;
pub use gateway::Endpoint;
pub use gateway::Response;
pub use http::Request;
pub use http::Server;

Modules§

gateway
What to do with a request. Pure, and the whole product lives here.
http
Enough HTTP/1.1 to be an honest gateway, and no more.