hearth-api
One OpenAI-compatible port in front of a fleet — where the HTTP status carries the diagnosis.
hearth: muse loading on 127.0.0.1:42499 …
hearth: whale not started — refused by the VRAM budget at declaration
hearth up on http://127.0.0.1:11434
POST /v1/chat/completions routed by the "model" field
GET /v1/models what is declared, and what is ready
GET /residency the truth the OpenAI shape cannot carry
Point any OpenAI client at it. Proxying is the boring part.
The part that is not boring
Every gateway can forward a request. What none of them do is answer honestly when they can't. A model still loading, a model the runtime evicted, a model whose GPU the host reclaimed, and a model that will never fit on this card all arrive at a caller as the same failed request.
hearth already knows which one it is. This crate turns that into a status a router can act on:
| situation | status | Retry-After |
retryable |
operator_fault |
|---|---|---|---|---|
| resident | proxied | — | — | — |
| warming | 503 | 5s | true | false |
| GPU reclaimed by host | 503 | 30s | true | false |
| evicted by runtime | 503 | 30s | true | true |
| process exited | 503 | 30s | true | true |
| will never fit here | 409 | none | false | false |
| not declared | 404 | none | false | — |
no model in body |
400 | none | false | — |
409, not 503, for a model that does not fit
This is the one that matters most and the easiest to get wrong. 503 means try again. A model that is short by 396 GiB on this card is arithmetically incapable of ever being served here — a retryable status turns that into a router hammering the box forever, which presents to everyone as a slow node.
No Retry-After header, deliberately. Setting one on a permanent condition invites exactly the retry storm the status code was chosen to prevent.
The body says whose fault it was
Same 503, opposite diagnosis. A router that reads only the status treats them alike — correct, both mean go elsewhere. One that reads the body learns who to score down, and that is the number that follows an operator around:
Each reason gets its own message rather than one keyed off the fault boolean. Telling an operator "this node is over-committed" because a process segfaulted sends them to inspect VRAM headroom that was never the problem — a wrong diagnosis is worse than a vague one.
Warming is progress, not failure
The caller is told how long, and decides for itself whether to wait. hearth never makes that call on someone else's behalf — a 32B materializing over a network fabric can legitimately take minutes, and killing it at a deadline turns a slow success into a fast failure.
/residency — what the OpenAI shape cannot express
The refused model is listed, not hidden. "It used to hold that model" is the first thing anyone says when a box gets slower, and the exact shortfall is the actionable number.
/v1/models carries the same truth in OpenAI's shape, under an additive hearth key that a plain client ignores.
/health is about the gateway, not the models. A gateway that reported unhealthy because a model was loading would be pulled from a load balancer for doing its job correctly.
Also speaks Ollama's paths
/api/chat, /api/generate and /api/tags are recognised, so a tool already pointed at Ollama does not need rewriting to try hearth.
Design
One lock, held only to decide. The gateway locks the fleet to make a routing decision and releases it before relaying. Holding it across a generation would stall every other request and the supervisor with it.
Streaming is relayed in chunks, flushed per chunk. Buffering would hold every token until the model finished — a live stream becomes a long silence followed by a wall of text, which is the most noticeable way a gateway ruins a chat UI.
No HTTP framework. Same call as the rest of this workspace: probe.rs writes a raw GET because a health check is a status line; hearth-pull drives curl because TLS and resume are a real library's job. This is a localhost server whose whole surface is read a request, write a response, relay bytes. A framework would bring an async runtime to do what std already does, and the parsing — the only part with a rule in it — lives here, pure and tested.
Loopback by default. A gateway on 0.0.0.0 hands anyone on the network an unauthenticated GPU. Put TLS in front of it if it needs to leave the box.
The rest of the workspace
| crate | what it does |
|---|---|
hearth-core |
the decision procedure — states, budget, routing |
hearth-resolve |
references to fetchable blobs, no network |
hearth-pull |
the bytes, digest-verified and recorded |
hearth-store |
the NEDB spine |
| hearth-api | this crate — the gateway |
hearth-serve |
the supervisor and the hearth CLI |
Built by Vex × Interchained
© Interchained LLC · BUSL-1.1 (converts to Apache-2.0 on 2030-08-27)