Skip to main content

Crate hearth_core

Crate hearth_core 

Source
Expand description

hearth — deterministic model residency.

Keep declared models warm, and tell the truth about which ones are.

Not an inference engine. llama.cpp and vLLM have spent years on kernels, samplers and tokenizers, and none of that is the problem. The problem is that no serving stack will promise a model stays loaded, and none of them can say why one stopped being. hearth is the supervisor above the runtime: it declares a set of models, refuses to declare more than the card holds, keeps them pinned, and reports residency as a named state with a named reason instead of an eventual timeout.

Two states here do not exist anywhere else, and they are the two that cost a night of debugging to tell apart:

  • Lost { reason: Evicted } — the runtime dropped it for VRAM.
  • Lost { reason: GpuDetached } — the host took the card away.

One is a capacity problem you own. The other is your provider’s, and no configuration you write will fix it. Today both look identical from the outside, which is why both get “fixed” repeatedly and neither goes away.

Re-exports§

pub use budget::plan;
pub use budget::total_ctx_tokens;
pub use budget::Budget;
pub use budget::Declared;
pub use budget::Plan;
pub use budget::Rejection;
pub use budget::GIB;
pub use fleet::Fleet;
pub use fleet::Route;
pub use fleet::Slot;
pub use gguf::read_kv_shape;
pub use gguf::KvShape;
pub use residency::LostReason;
pub use residency::Millis;
pub use residency::Observation;
pub use residency::Residency;

Modules§

budget
VRAM accounting. Refuse to overcommit instead of thrashing.
fleet
The fleet: every declared model, what state it is in, and what to tell a caller who wants one.
gguf
Read the four numbers a GGUF header carries that decide how big its KV cache gets. Nothing else in the file — not a tensor, not a token.
residency
What state a model is actually in, and how it got there.
sha256
SHA-256, because a digest you did not check is a download you did not verify.