# turbospark-server
High-performance local HTTP inference server (`turbospark-server` / `turbospark serve`) built on Axum. Implements OpenAI `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, and `/v1/images/generations`; Anthropic `/v1/messages` and `/v1/messages/count_tokens`; Ollama-compatible `/api/{tags,version,show,chat,generate,embeddings,embed}`; audio endpoints (`/v1/audio/{transcriptions,speech,generate,jobs,models}`); `/v1/models`; `/v1/metrics`; and `/health` liveness probes.
Supports both non-streaming JSON responses and Server-Sent Events (SSE) / NDJSON streaming formats. Uses zero-allocation translations between Anthropic and OpenAI protocols.
OpenAPI 3.1.0 specifications are in `docs/openapi/turbospark.openapi.yaml` and `docs/openapi/audio.openapi.yaml`.
## Purpose & Role
`turbospark-server` exposes local Apple Silicon inference to third-party tools, IDE extensions (Continue, Cursor, Copilot), agent frameworks (Claude Code, OpenCode, Hermes), and multimodal clients through standardized HTTP and WebSocket APIs.
## Binary Execution
```sh
# Launch server against a real .gturbo model install (macOS, release mode required)
cargo run --release -p turbospark-server --bin turbospark-server -- \
--model ~/models/gemma4.gturbo [--port 8080] [--bind loopback|tailnet]
# Launch server with round-robin multi-model concurrency pool
cargo run --release -p turbospark-server --bin turbospark-server -- \
--model ~/models/gemma4.gturbo --pool-size 2
# Attach distinct installs and route requests by model id or alias
cargo run --release -p turbospark-server --bin turbospark-server -- \
--model ~/models/gemma4.gturbo --model ~/models/qwen38-27b.gturbo
# Launch server with Speech-to-Text models (Whisper, Qwen3-ASR)
cargo run --release -p turbospark-server --bin turbospark-server -- \
--stt-model whisper-base-en --port 8080
# Launch audio-only server (no text generation model required)
cargo run --release -p turbospark-server --bin turbospark-server -- \
--stt-model whisper-base-en --tts-model ~/models/kokoro-82m --music-model minimax-music3-4bit
# Launch scripted server (canned completions, for client integration testing)
cargo run -p turbospark-server --bin turbospark-server -- <tokenizer-dir> [port]
```
## Key Modules
- `main.rs` & `args.rs`: Binary entry point and command-line flag parser (`--model`, repeatable for distinct installs, `--stt-model`, `--tts-model`, `--music-model`, `--bind`, `--port`, `--pool-size`, `--api-key`, `--max-queue-depth`).
- `handler/`: Core handlers for `/v1/chat/completions`, `/v1/models`, and `/health`, and generation orchestration (`plan`, `run_full`, `stream_blocking`).
- `audio/` & `audio_real.rs`: Dedicated audio worker threads, queues, and handlers for STT (`/v1/audio/transcriptions`, realtime WebSocket), TTS (`/v1/audio/speech`), Music (`/v1/audio/generate`), async jobs (`/v1/audio/jobs`), and model lifecycle (`/v1/audio/models`).
- `messages.rs`: Handler for Anthropic `/v1/messages` and `/v1/messages/count_tokens`, translating Anthropic messages to/from the OpenAI pipeline.
- `completions.rs`: Handler for legacy OpenAI `/v1/completions` (raw prompts without chat templates).
- `responses/`: Handler for the OpenAI `/v1/responses` endpoint.
- `embeddings.rs` & `encoder_model.rs`: `/v1/embeddings` and Ollama embedding routes backed by BERT/XLM-RoBERTa encoder models.
- `ollama.rs`: Ollama-compatible `/api/*` endpoints with NDJSON streaming.
- `guardrails.rs` & `guardrails/`: Tool-call rescue parser, schema repair, and validation retry loop.
- `registry.rs`: Model routing registry, including `PoolRegistry` for round-robin dispatch across multiple loaded model instances.
- `queue.rs`: Request FIFO generation queue managing concurrent request buffering and queue timeouts.
- `observe.rs`: Prometheus metrics collector and request latency instrumentation.
- `vision.rs`: Multimodal image URL decoding, download limits, and vision tensor injection.
- `bind.rs`: Network interface binding (loopback vs Tailscale tailnet).
- `auth.rs`: Bearer token and API key validation.
- `cancel.rs`: Client disconnect cancellation handling via cancellation tokens.
- `model.rs` & `real_model.rs`: `ChatModel` abstraction and `RealChatModel` driver.
- `response.rs`: Response envelopes and SSE chunk formatting.
## Development & Test Commands
```sh
# Run all server unit and integration tests
cargo test -p turbospark-server
# Test endpoint accessibility against a running server
curl -s http://127.0.0.1:8080/v1/models
```
## Tests
This crate contains 22 integration test suites in `tests/`:
- Protocol compatibility: `chat_completions.rs`, `completions.rs`, `messages.rs`, `responses.rs`, `ollama.rs`, `embeddings_api.rs`.
- Concurrency and queues: `generation_queue.rs`, `registry.rs`, `cancellation.rs`.
- Security and transport: `auth.rs`, `streaming_error_framing.rs`.
- Tool calling and guardrails: `guardrails.rs`, `system_prompt.rs`, `harmony_channels.rs`, `reasoning_channels.rs`.
- Multimodal and real model tests: `images.rs`, `real_backend.rs`, `observe.rs`.
- Audio API: `audio_api.rs`.
## Crate Gotchas
1. **Serialized Real Backend & Concurrency Pools**: A single `RealChatModel` instance processes requests serially because Metal forward passes require mutable GPU buffers and KV cache state. For concurrent request processing, configure `--pool-size N` to instantiate multiple runner instances managed by `PoolRegistry`.
2. **Anthropic SSE Framing Differences**: Anthropic SSE streams use explicit `event:` header lines (e.g. `content_block_delta`, `message_delta`) and terminate with a `message_stop` event rather than the OpenAI `data: [DONE]` sentinel.
3. **Tailnet Binding Security**: Setting `--bind tailnet` binds exclusively to the Tailscale IPv4 carrier grade interface (100.64.0.0/10) with mandatory API key authentication.
4. **Speech Installs Refused on `--model`**: Speech (Whisper, Qwen3-ASR) installs carry a typed speech receipt and are refused by `turbospark-server --model`, which serves `.gturbo` text and vision models. Attach speech models via `--stt-model <install-dir|alias>` (repeatable), TTS models via `--tts-model`, and music models via `--music-model`. With no `--model`, the server runs in audio-only mode. Audio models can also be dynamically attached at runtime via `POST /v1/audio/models`.