Crate memra_server 

memra-server (BASE-4): a minimal OpenAI-ish HTTP server that serves 2-4 concurrent agents across DIFFERENT models on one endpoint via a single GPU worker thread + step-interleave scheduler.

Architecture (see worker.rs): axum runs on a tokio runtime; ONE dedicated std::thread owns the Engine + every loaded HybridModel (CUDA context is thread-affine). Handlers submit Cmds over a std mpsc channel and receive tokens back over a per-request tokio mpsc channel.

Endpoints (the full set — router() below is the authority): GET /health, GET /livez -> the SAME handler (health_live): INFERENCE liveness, not process liveness. {“status”:“ok”|“draining”|“unhealthy”, “models”:[…], “worker”:{phase, beat_age_ms, tick_max_ms, stall_threshold_ms, generation, xid_warnings}} + a top-level “detail” on a red. Draining stays 200; dead / GPU-faulted / stalled / loading is 503 (serve-hardening 2026-08-06). GET /readyz -> routability, same payload shape with “status”:“ready”|“not_ready”. Unready is NOT a restart request — draining and loading are healthy-but-unroutable. GET /models -> {“data”:[{“id”:name},…]} (OpenAI-ish); ?schema=openrouter -> Provider Monitor schema 2.4, ?schema=openmodels -> OpenModels provider feed. GET /v1/models -> existing catalog-style model list (context_length, architecture, pricing stub, top_provider; serve-tail). GET /metrics -> flat serving counters + step latency percentiles. GET /yield/metrics -> per-lane x-lane QoS counters + engine-truth step p50/p99 (lane/qos-p95 2026-08-02). POST /v1/completions -> {model,prompt|prompt_ids,max_tokens,temperature?,top_p?,top_k?, seed?,stop?,chat?,stream?,cache_salt?}. stream=true => SSE token-by-token; else a single JSON {text,tokens,stop_reason}. POST /v1/chat/completions -> OpenAI chat messages rendered by the GGUF chat template; OpenAI message/chunk response shapes. tools/tool_choice (auto|none) + role:“tool” turns render through the template’s own branch; emitted <tool_call> blocks parse into OpenAI tool_calls (+“tool_calls” finish); reasoning_effort/reasoning map onto the template’s think switch (serve-tools lane, 2026-08-02).

CONFIG: MEMRA_MODELS=“name=/path.gguf[+/draft.gguf],name2=hf:owner/repo,name3=/hf_ckpt_dir” (comma-separated; +draft.gguf attaches that model’s regime draft — docs/DRAFT-REGIME.md). A model path may be a GGUF file OR an HF safetensors checkpoint directory (config.json + model.safetensors[.index.json] — the run-safetensors load path; serve-st lane 2026-08-04). Defaults to the BASE-4 test pair (main=27B, judge=9B) if unset. MEMRA_ADDR sets the bind addr.

LIFECYCLE: SIGTERM = graceful drain (gap-scan F11) — new completion requests 503 with Retry-After, /health reports “draining”, in-flight requests (streams included) finish up to MEMRA_DRAIN_S (default 30s), then the process exits 0. Completion responses carry X-RateLimit-Limit/-Remaining/-Reset (concurrency-slot semantics; gap-scan F12).

Modules

auth

x-lane QoS (lane/dl-metering gate, QoS-only extraction 2026-08-02): lane types, SLO admission policy, engine-truth step stats live in the memra-lanes crate so out-of-process controllers (the sidecar shape) can share them.

metering

The admission/accounting seam: the server admits, denies, and reports counts; what admission MEANS — budgets, prices, tenancy policy — is a deployment concern, supplied behind metering::Metering through ServerWiring. The stock binary ships NO accounting (only the engine is open; the business tier lives in the deployment’s own binary — engine-billing-extraction-20260829, owner razor 2026-08-29: “only engine is open, business is private”). The admission/accounting seam (lane engine-billing-extraction-20260829).

Structs

PurgeHandle

Purge one tenant’s parked KV state (the engine half of a deployment admin /admin/tenants/{tenant}/purge; lane/kv-tenancy-compaction-20260831, tiering spec §0.5). Contract notes for the deployment surface: the path parameter is {tenant} (the keyring tenant id, the same string --gen-key <tenant> took), never {tenant_id}; fire it from key revocation AND tenant deletion; a report with device_pinned_left > 0 means in-flight sessions still lease device entries in the tenant’s namespaces, so re-fire after the drain. Cloneable, same lifetime contract as TrimHandle: drop it on the shutdown signal.

RuntimeHandles

Runtime handles handed to ServerWiring::on_ready — the narrow set of engine-runtime operations a deployment-side admin surface needs.

ServerWiring

Deployment wiring for a custom binary. serve_main is exactly serve_with(ServerWiring::reference()); a deployment-owned binary substitutes its own metering and hooks the runtime handles it needs.

TrimHandle

Ask the worker to trim its pools (the engine half of /admin/trim). Cloneable; answers with the worker’s own trim report.

Functions

serve_main

The whole server as a library entry point (BASE-4 stays: this crate is the async-only seam; the bin in src/main.rs is one line deep). Public so a deployment-owned binary can wrap the same server with its own wiring.

serve_with