memra-server (BASE-4): a minimal OpenAI-ish HTTP server that serves 2-4 concurrent agents across DIFFERENT models on one endpoint via a single GPU worker thread + step-interleave scheduler.
Architecture (see worker.rs): axum runs on a tokio runtime; ONE dedicated std::thread owns the
Engine + every loaded HybridModel (CUDA context is thread-affine). Handlers submit Cmds over a
std mpsc channel and receive tokens back over a per-request tokio mpsc channel.
Endpoints:
GET /health -> {"status":"ok"|"draining","models":[...]}
GET /models -> {"data":[{"id":name},...]} (OpenAI-ish)
GET /v1/models -> OR-schema model list (context_length, architecture,
pricing stub, top_provider; serve-tail 2026-08-04).
GET /metrics -> flat serving counters + step latency percentiles.
POST /v1/completions -> {model,prompt|prompt_ids,max_tokens,temperature?,top_p?,top_k?,
seed?,stop?,chat?,stream?,cache_salt?}. stream=true => SSE
token-by-token; else a single JSON {text,tokens,stop_reason}.
POST /v1/chat/completions -> OpenAI chat messages rendered by the GGUF chat template;
OpenAI message/chunk response shapes. tools/tool_choice
(auto|none) + role:"tool" turns render through the
template's own branch; emitted <tool_call> blocks
parse into OpenAI tool_calls (+"tool_calls" finish);
reasoning_effort/reasoning map onto the template's
think switch (serve-tools lane, 2026-08-02).
CONFIG: MEMRA_MODELS="name=/path.gguf[+/draft.gguf],name2=hf:owner/repo,name3=/hf_ckpt_dir"
(comma-separated; +draft.gguf attaches that model's regime draft — docs/DRAFT-REGIME.md).
A model path may be a GGUF file OR an HF safetensors checkpoint directory
(config.json + model.safetensors[.index.json] — the run-safetensors load path; serve-st
lane 2026-08-04). Defaults to the BASE-4 test pair (main=27B, judge=9B) if unset.
MEMRA_ADDR sets the bind addr.
LIFECYCLE: SIGTERM = graceful drain (gap-scan F11) — new completion requests 503 with Retry-After, /health reports "draining", in-flight requests (streams included) finish up to MEMRA_DRAIN_S (default 30s), then the process exits 0. Completion responses carry X-RateLimit-Limit/-Remaining/-Reset (concurrency-slot semantics; gap-scan F12).