Skip to main content

Crate ramvamp_server

Crate ramvamp_server 

Source
Expand description

OpenAI-compatible HTTP server for the ramvamp runtime.

The crate is layered so that the wire contract can be tested without a 30B model on disk, and so that the timing of a stream can be tested without a socket. Every rule below is asserted by a unit test that runs in milliseconds, and a streaming sequence is pinned byte for byte.

§The pure layer: shapes and rules

  • request — the POST /v1/chat/completions body, including the array-form content that clients send routinely and a naive server silently discards.
  • response — chat.completion, chat.completion.chunk, the /v1/models listing and the /health body.
  • sse — SSE payload construction: data: {json}\n\n and the bare [DONE] sentinel.
  • prompt — mapping messages onto the vendored ChatML renderer, through the sanitizing encoder because the input is untrusted.
  • toolcall — the return leg: recovering tool_calls from the <tool_call> text the model writes, preserving the argument bytes so the client’s echo of them re-renders into the block the model emitted. Twice over, because a stream cannot wait for the whole reply: extract for the buffered path and CallStream for the streaming one, the latter keyed off the marker token ids so a boundary cannot fall between two deltas.
  • error — one typed error per refusal, each knowing its status, body and headers.

§The engine layer

  • engine — the Engine trait, which is the seam a stub token source substitutes into, plus ModelEngine: model, forward state and tokenizer on one thread, with the KV cache reused across requests by longest-common-prefix match. The trap in that match — and why the match is against generated ids and never against a re-render of the transcript — is documented there, because getting it wrong is silent.

§The transport layer

  • wire — HTTP/1.1 chunked framing written by hand, because tiny_http::Response buffers a stream to completion and hides client disconnects.
  • http — routing, body limits and the accept loop. Binds loopback only.

§What it is sized for

One session, on loopback, in front of a runtime that decodes at about 2 tok/s. That shapes decisions that would look wrong on a hosted API: n > 1 is refused instead of approximated, a busy server answers 503 rather than 429, and Retry-After is a small fiction rather than an honest wait. Each of those has its reasoning recorded next to the code, because each of them looks like a bug until you know why.

The same rule governs the parameters this build cannot implement rather than cannot afford: a non-empty stop and a tool_choice that forces a particular call are 400s, because nothing downstream applies either one and a 200 that quietly ignored them would be indistinguishable from a correct answer. tools themselves are served — see prompt.

Re-exports§

pub use engine::Completion;
pub use engine::Engine;
pub use engine::EngineConfig;
pub use engine::ModelEngine;
pub use engine::NullSink;
pub use engine::Plan;
pub use engine::StreamError;
pub use engine::TokenSink;
pub use engine::common_prefix;
pub use engine::hush_stream_aborts;
pub use error::ErrorDetail;
pub use error::ErrorResponse;
pub use error::RETRY_HEADERS;
pub use error::ServerError;
pub use http::BODY_BYTES_PER_CONTEXT_TOKEN;
pub use http::MIN_BODY_BYTES;
pub use http::ServeConfig;
pub use http::ServeError;
pub use http::max_body_bytes_for_context;
pub use http::serve;
pub use http::serve_with;
pub use prompt::Prompt;
pub use prompt::check_context;
pub use request::ChatCompletionRequest;
pub use request::Content;
pub use request::ContentPart;
pub use request::Message;
pub use request::MessageRole;
pub use request::StreamOptions;
pub use request::StringOrArray;
pub use request::ToolCall;
pub use response::ChatCompletion;
pub use response::ChatCompletionChunk;
pub use response::ChunkBuilder;
pub use response::Delta;
pub use response::FinishReason;
pub use response::Health;
pub use response::ModelList;
pub use response::ResponseMessage;
pub use response::ToolCallDelta;
pub use response::Usage;
pub use sse::DONE_SENTINEL;
pub use sse::sse_data;
pub use sse::sse_done;
pub use sse::sse_event;
pub use toolcall::CallStream;
pub use toolcall::Extracted;
pub use toolcall::ParsedCall;
pub use toolcall::Step;
pub use toolcall::call_id;
pub use toolcall::extract;
pub use toolcall::wire_call;
pub use toolcall::wire_calls;

Modules§

engine
The thing that turns a request into tokens, and the seam that lets the transport be tested without one.
error
Typed request/response errors and their HTTP mapping.
http
The HTTP transport: one thread, one socket, one request at a time.
prompt
Turning a request’s messages into the token ids the engine prefills.
request
POST /v1/chat/completions request shapes.
response
/v1/chat/completions response shapes, streaming and not, plus the /v1/models and /health bodies.
sse
Server-sent-event payload construction.
toolcall
Recovering structured tool_calls from the reply text the model wrote.
wire
Hand-written HTTP/1.1 chunked framing for the streaming path.