memra-server (BASE-4): a minimal OpenAI-ish HTTP server that serves 2-4 concurrent agents across DIFFERENT models on one endpoint via a single GPU worker thread + step-interleave scheduler.
Architecture (see worker.rs): axum runs on a tokio runtime; ONE dedicated std::thread owns the
Engine + every loaded HybridModel (CUDA context is thread-affine). Handlers submit Cmds over a
std mpsc channel and receive tokens back over a per-request tokio mpsc channel.
Endpoints (the full set — router() below is the authority):
GET /health, GET /livez -> the SAME handler (health_live): INFERENCE liveness, not
process liveness. {"status":"ok"|"draining"|"unhealthy",
"models":[...], "worker":{phase, beat_age_ms, tick_max_ms,
stall_threshold_ms, generation, xid_warnings}} + a
top-level "detail" on a red. Draining stays 200; dead /
GPU-faulted / stalled / loading is 503 (serve-hardening
2026-08-06).
GET /readyz -> routability, same payload shape with
"status":"ready"|"not_ready". Unready is NOT a restart
request — draining and loading are healthy-but-unroutable.
GET /models -> {"data":[{"id":name},...]} (OpenAI-ish);
?schema=openrouter -> Provider Monitor schema 2.4.
GET /v1/models -> existing catalog-style model list (context_length,
architecture, pricing stub, top_provider; serve-tail).
GET /metrics -> flat serving counters + step latency percentiles.
GET /yield/metrics -> per-lane x-lane QoS counters + engine-truth step p50/p99
(lane/qos-p95 2026-08-02).
POST /v1/completions -> {model,prompt|prompt_ids,max_tokens,temperature?,top_p?,top_k?,
seed?,stop?,chat?,stream?,cache_salt?}. stream=true => SSE
token-by-token; else a single JSON {text,tokens,stop_reason}.
POST /v1/chat/completions -> OpenAI chat messages rendered by the GGUF chat template;
OpenAI message/chunk response shapes. tools/tool_choice
(auto|none) + role:"tool" turns render through the
template's own branch; emitted <tool_call> blocks
parse into OpenAI tool_calls (+"tool_calls" finish);
reasoning_effort/reasoning map onto the template's
think switch (serve-tools lane, 2026-08-02).
CONFIG: MEMRA_MODELS="name=/path.gguf[+/draft.gguf],name2=hf:owner/repo,name3=/hf_ckpt_dir"
(comma-separated; +draft.gguf attaches that model's regime draft — docs/DRAFT-REGIME.md).
A model path may be a GGUF file OR an HF safetensors checkpoint directory
(config.json + model.safetensors[.index.json] — the run-safetensors load path; serve-st
lane 2026-08-04). Defaults to the BASE-4 test pair (main=27B, judge=9B) if unset.
MEMRA_ADDR sets the bind addr.
LIFECYCLE: SIGTERM = graceful drain (gap-scan F11) — new completion requests 503 with Retry-After, /health reports "draining", in-flight requests (streams included) finish up to MEMRA_DRAIN_S (default 30s), then the process exits 0. Completion responses carry X-RateLimit-Limit/-Remaining/-Reset (concurrency-slot semantics; gap-scan F12).