Expand description
OpenAI-compatible HTTP server for the ramvamp runtime.
The crate is layered so that the wire contract can be tested without a 30B model on disk, and so that the timing of a stream can be tested without a socket. Every rule below is asserted by a unit test that runs in milliseconds, and a streaming sequence is pinned byte for byte.
§The pure layer: shapes and rules
request— thePOST /v1/chat/completionsbody, including the array-formcontentthat clients send routinely and a naive server silently discards.response—chat.completion,chat.completion.chunk, the/v1/modelslisting and the/healthbody.sse— SSE payload construction:data: {json}\n\nand the bare[DONE]sentinel.prompt— mapping messages onto the vendored ChatML renderer, through the sanitizing encoder because the input is untrusted.toolcall— the return leg: recoveringtool_callsfrom the<tool_call>text the model writes, preserving the argument bytes so the client’s echo of them re-renders into the block the model emitted. Twice over, because a stream cannot wait for the whole reply:extractfor the buffered path andCallStreamfor the streaming one, the latter keyed off the marker token ids so a boundary cannot fall between two deltas.error— one typed error per refusal, each knowing its status, body and headers.
§The engine layer
engine— theEnginetrait, which is the seam a stub token source substitutes into, plusModelEngine: model, forward state and tokenizer on one thread, with the KV cache reused across requests by longest-common-prefix match. The trap in that match — and why the match is against generated ids and never against a re-render of the transcript — is documented there, because getting it wrong is silent.
§The transport layer
wire— HTTP/1.1 chunked framing written by hand, becausetiny_http::Responsebuffers a stream to completion and hides client disconnects.http— routing, body limits and the accept loop. Binds loopback only.
§What it is sized for
One session, on loopback, in front of a runtime that decodes at about 2
tok/s. That shapes decisions that would look wrong on a hosted API:
n > 1 is refused instead of approximated, a busy server answers 503 rather
than 429, and Retry-After is a small fiction rather than an honest wait.
Each of those has its reasoning recorded next to the code, because each of
them looks like a bug until you know why.
The same rule governs the parameters this build cannot implement rather
than cannot afford: a non-empty stop and a tool_choice that forces a
particular call are 400s, because nothing downstream applies either one and
a 200 that quietly ignored them would be indistinguishable from a correct
answer. tools themselves are served — see prompt.
Re-exports§
pub use engine::Completion;pub use engine::Engine;pub use engine::EngineConfig;pub use engine::ModelEngine;pub use engine::NullSink;pub use engine::Plan;pub use engine::StreamError;pub use engine::TokenSink;pub use engine::common_prefix;pub use engine::hush_stream_aborts;pub use error::ErrorDetail;pub use error::ErrorResponse;pub use error::RETRY_HEADERS;pub use error::ServerError;pub use http::BODY_BYTES_PER_CONTEXT_TOKEN;pub use http::MIN_BODY_BYTES;pub use http::ServeConfig;pub use http::ServeError;pub use http::max_body_bytes_for_context;pub use http::serve;pub use http::serve_with;pub use prompt::Prompt;pub use prompt::check_context;pub use request::ChatCompletionRequest;pub use request::Content;pub use request::ContentPart;pub use request::Message;pub use request::MessageRole;pub use request::StreamOptions;pub use request::StringOrArray;pub use request::ToolCall;pub use response::ChatCompletion;pub use response::ChatCompletionChunk;pub use response::ChunkBuilder;pub use response::Delta;pub use response::FinishReason;pub use response::Health;pub use response::ModelList;pub use response::ResponseMessage;pub use response::ToolCallDelta;pub use response::Usage;pub use sse::DONE_SENTINEL;pub use sse::sse_data;pub use sse::sse_done;pub use sse::sse_event;pub use toolcall::CallStream;pub use toolcall::Extracted;pub use toolcall::ParsedCall;pub use toolcall::Step;pub use toolcall::call_id;pub use toolcall::extract;pub use toolcall::wire_call;pub use toolcall::wire_calls;
Modules§
- engine
- The thing that turns a request into tokens, and the seam that lets the transport be tested without one.
- error
- Typed request/response errors and their HTTP mapping.
- http
- The HTTP transport: one thread, one socket, one request at a time.
- prompt
- Turning a request’s
messagesinto the token ids the engine prefills. - request
POST /v1/chat/completionsrequest shapes.- response
/v1/chat/completionsresponse shapes, streaming and not, plus the/v1/modelsand/healthbodies.- sse
- Server-sent-event payload construction.
- toolcall
- Recovering structured
tool_callsfrom the reply text the model wrote. - wire
- Hand-written HTTP/1.1 chunked framing for the streaming path.