hearth-api 0.5.0

An OpenAI-compatible port in front of a hearth fleet, where the HTTP status carries the diagnosis.
Documentation

hearth-api

One OpenAI-compatible port in front of a fleet — where the HTTP status carries the diagnosis.

crates.io docs.rs

hearth up --model muse=/models/muse.gguf:20 \
          --model deepseek=/models/deepseek.gguf:20 \
          --total-gib 48 --port 11434
hearth: muse loading on 127.0.0.1:42499 …
hearth: whale not started — refused by the VRAM budget at declaration
hearth up on http://127.0.0.1:11434
  POST /v1/chat/completions   routed by the "model" field
  GET  /v1/models             what is declared, and what is ready
  GET  /residency             the truth the OpenAI shape cannot carry

Point any OpenAI client at it. Proxying is the boring part.

The part that is not boring

Every gateway can forward a request. What none of them do is answer honestly when they can't. A model still loading, a model the runtime evicted, a model whose GPU the host reclaimed, and a model that will never fit on this card all arrive at a caller as the same failed request.

hearth already knows which one it is. This crate turns that into a status a router can act on:

situation status Retry-After retryable operator_fault
resident proxied — — —
warming 503 5s true false
GPU reclaimed by host 503 30s true false
evicted by runtime 503 30s true true
process exited 503 30s true true
will never fit here 409 none false false
not declared 404 none false —
no model in body 400 none false —

409, not 503, for a model that does not fit

This is the one that matters most and the easiest to get wrong. 503 means try again. A model that is short by 396 GiB on this card is arithmetically incapable of ever being served here — a retryable status turns that into a router hammering the box forever, which presents to everyone as a slow node.

$ curl -s -D- -X POST localhost:11434/v1/chat/completions \
    -d '{"model":"whale","messages":[]}'
HTTP/1.1 409 Conflict
{"error": {
  "message": "whale does not fit on this host — short by 396.0 GiB. This is permanent
              until the declaration or the hardware changes, so do not retry here.",
  "type": "model_not_admitted",
  "short_bytes": 425201762304,
  "retryable": false
}}

No Retry-After header, deliberately. Setting one on a permanent condition invites exactly the retry storm the status code was chosen to prevent.

The body says whose fault it was

Same 503, opposite diagnosis. A router that reads only the status treats them alike — correct, both mean go elsewhere. One that reads the body learns who to score down, and that is the number that follows an operator around:

{"type": "model_lost", "reason": "gpudetached", "operator_fault": false,
 "message": "muse lost its GPU — the host reclaimed the card; this is not the operator's doing"}

{"type": "model_lost", "reason": "evicted", "operator_fault": true,
 "message": "muse was evicted by the runtime to free VRAM — this node is over-committed"}

Each reason gets its own message rather than one keyed off the fault boolean. Telling an operator "this node is over-committed" because a process segfaulted sends them to inspect VRAM headroom that was never the problem — a wrong diagnosis is worse than a vague one.

Warming is progress, not failure

{"type": "model_warming", "loading_for_ms": 20000, "retryable": true,
 "operator_fault": false,
 "message": "muse is loading (20s so far) — this is progress, not a failure"}

The caller is told how long, and decides for itself whether to wait. hearth never makes that call on someone else's behalf — a 32B materializing over a network fabric can legitimately take minutes, and killing it at a deadline turns a slow success into a fast failure.

/residency — what the OpenAI shape cannot express

$ curl -s localhost:11434/residency
{
  "committed_bytes": 2147483648,
  "free_bytes": 4294967296,
  "report": "2.0 / 6.0 GiB held  (2 declared, 1 admitted)\n  muse   resident for 25s\n  whale  not admitted — short by 396.0 GiB",
  "models": [
    {"model": "muse",  "ready": true,  "operator_fault": false, "short_bytes": 0},
    {"model": "whale", "ready": false, "operator_fault": false, "short_bytes": 425201762304}
  ]
}

The refused model is listed, not hidden. "It used to hold that model" is the first thing anyone says when a box gets slower, and the exact shortfall is the actionable number.

/v1/models carries the same truth in OpenAI's shape, under an additive hearth key that a plain client ignores.

/health is about the gateway, not the models. A gateway that reported unhealthy because a model was loading would be pulled from a load balancer for doing its job correctly.

Also speaks Ollama's paths

/api/chat, /api/generate and /api/tags are recognised, so a tool already pointed at Ollama does not need rewriting to try hearth.

Design

One lock, held only to decide. The gateway locks the fleet to make a routing decision and releases it before relaying. Holding it across a generation would stall every other request and the supervisor with it.

Streaming is relayed in chunks, flushed per chunk. Buffering would hold every token until the model finished — a live stream becomes a long silence followed by a wall of text, which is the most noticeable way a gateway ruins a chat UI.

No HTTP framework. Same call as the rest of this workspace: probe.rs writes a raw GET because a health check is a status line; hearth-pull drives curl because TLS and resume are a real library's job. This is a localhost server whose whole surface is read a request, write a response, relay bytes. A framework would bring an async runtime to do what std already does, and the parsing — the only part with a rule in it — lives here, pure and tested.

Loopback by default. A gateway on 0.0.0.0 hands anyone on the network an unauthenticated GPU. Put TLS in front of it if it needs to leave the box.

The rest of the workspace

crate what it does
hearth-core the decision procedure — states, budget, routing
hearth-resolve references to fetchable blobs, no network
hearth-pull the bytes, digest-verified and recorded
hearth-store the NEDB spine
hearth-api this crate — the gateway
hearth-serve the supervisor and the hearth CLI

Built by Vex × Interchained

© Interchained LLC · BUSL-1.1 (converts to Apache-2.0 on 2030-08-27)