mothership 0.0.56

# Mothership

Process supervisor with HTTP exposure. Launches your fleet, routes traffic, runs WASM plugins.

## Features

- **Fleet Management**: Launch and monitor ships and bays with dependency ordering
- **Prelaunch Jobs**: Run migrations and setup tasks before any ship starts
- **Flagship Coordination**: Multi-server leader election for prelaunch (static or PostgreSQL)
- **Uplinks**: Pre-flight connectivity checks for external dependencies (databases, caches)
- **Bays (Docking Protocol)**: WebSocket multiplexing over Unix sockets for real-time apps
- **Named Binds**: Multiple listeners (http, https, ws) routed via TCP or Unix sockets
- **Static File Serving**: Serve static assets with configurable path prefix
- **Response Compression**: Automatic gzip/deflate/br compression
- **CORS Preflight Cache**: Cache backend OPTIONS responses per-origin
- **Health Checks**: HTTP-based monitoring with retry and circuit breakers
- **Crash Loop Protection**: Exponential backoff for failing ships
- **Resilience Patterns**: Circuit breakers, retries, and optional rate limiting
- **Memory Monitoring**: Per-ship RSS/virtual memory metrics (Linux, macOS, FreeBSD)
- **TUI Dashboard**: Real-time fleet status with logs (`--tui`)
- **WASM Payloads**: Request/response processing with sandboxing
- **Prometheus Metrics**: Ship status, restarts, request counts at `/metrics`
- **Tag Filtering**: `--only web` or `--except workers` for selective launching
- **Base Templates**: Reduce config duplication
- **Graceful Shutdown**: Dependency-aware termination

## Installation

```bash
cargo install mothership
```

Or from source:
```bash
git clone https://github.com/seuros/mothership.git
cd mothership
cargo build --release
```

### Cargo Features

Mothership uses optional features to minimize binary size:

| Feature | Size Impact | Description |
|---------|-------------|-------------|
| `tui` | +0.3 MB | TUI dashboard (`--tui` flag) |
| `wasm` | +8.4 MB | WASM payload processing |
| `tokio-postgres` | +0.5 MB | PostgreSQL flagship election |

Default build (no features): ~10.4 MB

```bash
# Minimal build (no TUI, no WASM)
cargo build --release

# With TUI dashboard
cargo build --release --features tui

# With WASM payloads
cargo build --release --features wasm

# Full build (all features)
cargo build --release --features "tui,wasm,tokio-postgres"
```

## Quick Start

```bash
# Initialize a new manifest
mothership init

# Validate configuration
mothership clearance

# View routing chart
mothership chart

# Run the fleet
mothership

# Run with TUI dashboard
mothership --tui
```

## Container Runtime Note (Linux PID 1)

When `mothership` runs as PID 1 on Linux, it enables an internal init shim:
- PID 1 becomes a minimal reaper/forwarder
- the real mothership runtime runs as a child process (PID > 1)

This avoids Tokio child-process supervision races under PID 1.
You should still prefer an external init in containers:

```bash
# Docker
docker run --init ...
```

```yaml
# docker-compose.yml
services:
  mothership:
    init: true
```

If you use another runtime, an equivalent `tini`/init wrapper is required.

## Configuration

Create `ship-manifest.toml`:

```toml
# Global configuration
[mothership]
metrics_port = 9090
compression = true

# Static files (served before routing to ships, with fallthrough)
# Longest prefix wins; if file not found, falls through to routes
[[mothership.static_dirs]]
path = "./public"
prefix = "/static"

# Named binds - external listeners
[mothership.bind]
http = "0.0.0.0:80"
https = "0.0.0.0:443"
ws = "0.0.0.0:8080"

# Web ships
[[fleet.web]]
name = "app"
command = "ruby"
args = ["app.rb"]
bind = "tcp://127.0.0.1:3000"
healthcheck = "/health"
routes = [
  { bind = "http", pattern = "/.*" },
  { bind = "https", pattern = "/.*" },
]

# Background workers
[[fleet.workers]]
name = "sidekiq"
command = "bundle"
args = ["exec", "sidekiq"]
critical = false

# One-shot jobs
[[fleet.jobs]]
name = "migrate"
command = "rails"
args = ["db:migrate"]
oneshot = true

# Docked bays (WebSocket multiplexing)
[[bays.websocket]]
name = "orbitcast"
command = "./orbitcast"
routes = [{ bind = "ws", pattern = "/ws" }]
config = { redis_url = "redis://localhost:6379" }

# WASM payloads (optional)
[[modules]]
name = "rate_limiter"
wasm = "./modules/rate_limiter.wasm"
routes = ["/api/.*"]
phase = "request"
```

## Ships vs Bays

**Ships** are traditional processes that bind to a port (TCP or Unix socket). Mothership proxies HTTP/WebSocket traffic to them.

**Bays** use the docking protocol for WebSocket multiplexing. Multiple client connections are multiplexed over a single Unix socket. Ideal for:
- Real-time apps (chat, notifications)
- Reducing connection overhead
- Processes that don't need their own HTTP server

### Ship Configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `name` | string | required | Unique identifier (ASCII only, no spaces) |
| `command` | string | required | Command to execute |
| `args` | string[] | `[]` | Command arguments |
| `bind` | string | - | Internal bind address (`tcp://host:port` or `unix:///path`) |
| `healthcheck` | string | - | Health check endpoint path |
| `routes` | array | `[]` | HTTP routes (see below) |
| `depends_on` | string[] | `[]` | Ships/bays to wait for before starting |
| `env` | table | `{}` | Environment variables |
| `critical` | bool | `true` | Crash kills entire fleet |
| `oneshot` | bool | `false` | Run once and exit |
| `tags` | string[] | `[]` | Tags for filtering (`--only`, `--except`) |

### Bay Configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `name` | string | required | Unique identifier (ASCII only, no spaces) |
| `command` | string | required | Command to execute |
| `args` | string[] | `[]` | Command arguments |
| `routes` | array | `[]` | WebSocket routes |
| `depends_on` | string[] | `[]` | Ships/bays to wait for |
| `env` | table | `{}` | Environment variables |
| `config` | table | `{}` | Config passed via docking protocol |
| `critical` | bool | `true` | Crash kills entire fleet |
| `tags` | string[] | `[]` | Tags for filtering |

### Route Configuration

Routes map mothership binds to ships/bays:

```toml
# Object format
routes = [
  { bind = "http", pattern = "/api/.*" },
  { bind = "ws", pattern = "/cable" },
]

# Shorthand format
routes = ["http:/api/.*", "ws:/cable"]

# With path stripping
routes = [
  { bind = "http", pattern = "/api/.*", strip_prefix = "/api" },
]

# With User-Agent filtering (route LLM traffic to markdown-only backend)
routes = [
  { bind = "http", pattern = "/.*", ua_filter = "llm" },
]
```

### User-Agent Routing

Route traffic to different backends based on User-Agent:

```toml
# Browser-only route (Chromium, Firefox, Safari)
[[fleet.web]]
name = "app"
command = "rails"
routes = [{ bind = "http", pattern = "/.*", ua_filter = "browser" }]

# LLM-only route (Claude, GPT, Perplexity, etc.) - serve markdown
[[fleet.web]]
name = "markdown-api"
command = "rails"
routes = [{ bind = "http", pattern = "/.*", ua_filter = "llm" }]

# Bot/crawler route
[[fleet.web]]
name = "static-cache"
command = "nginx"
routes = [{ bind = "http", pattern = "/.*", ua_filter = "bot" }]
```

**Available filters:**
- `browser` - Chromium, Firefox, Safari browsers
- `chromium`, `firefox`, `safari` - Specific browser kinds
- `llm` - Claude/Anthropic agents
- `bot` - Bots, crawlers, curl, wget, etc.
- `~pattern` - Custom regex pattern (e.g., `~MyAgent.*`)

Routes are matched in declaration order. Put specific UA filters before catch-all routes.

### Shields (HTTP Fingerprinting)

Mothership computes Ja4H fingerprints for incoming requests. Ja4H fingerprints HTTP header order and values to detect bots/headless browsers even when they spoof User-Agent.

Fingerprints are logged with each request:
```
DEBUG method=GET path=/ ua_kind=Chromium shields=Some("ge11nn06enus_...")
```

Future: Shield-based routing to block or redirect suspicious fingerprints.

### Bind Formats

```toml
# TCP with explicit prefix
bind = "tcp://127.0.0.1:3000"

# TCP without prefix
bind = "0.0.0.0:8080"

# Port only (defaults to 127.0.0.1)
bind = "3000"

# Unix socket
bind = "unix:///tmp/app.sock"
```

### Mothership Binds with PROXY Protocol

When running behind a load balancer (AWS ELB/ALB, Cloudflare, HAProxy, nginx), enable PROXY protocol to preserve client IPs:

```toml
[mothership.bind]
http = "0.0.0.0:80"                                      # Direct access
https = { addr = "0.0.0.0:443", proxy_protocol = true }  # Behind LB
```

The `proxy_protocol` option enables HAProxy PROXY protocol v1/v2 with auto-detection (works with or without the protocol header).

Configure your load balancer to send PROXY protocol:
- **AWS ELB/NLB**: Enable "Proxy Protocol v2" in target group settings
- **Cloudflare Spectrum**: Enable "Proxy Protocol" in application settings
- **HAProxy**: Add `send-proxy` or `send-proxy-v2` to server line
- **nginx**: Add `proxy_protocol on` to upstream

## Docking Protocol

Bays communicate with mothership via Unix sockets using a binary protocol:

| Message | Direction | Purpose |
|---------|-----------|---------|
| `Dock` | Bay → Mothership | Bay ready, sends version |
| `Moored` | Mothership → Bay | Docking confirmed, sends config |
| `Boarding` | Mothership → Bay | New WebSocket client connected |
| `Disembark` | Mothership → Bay | Client disconnected |
| `Cargo` | Bidirectional | WebSocket data payload |

Environment variables provided to bays:
- `MS_PID` - Mothership process ID
- `MS_SHIP` - Bay name
- `MS_SOCKET_DIR` - Socket directory
- `MS_SOCKET_PATH` - Full socket path
- `MS_BAY_TYPE` - Bay type (e.g., "websocket")

## Commands

```bash
# Run fleet (default)
mothership
mothership run
mothership run -c /path/to/manifest.toml

# Run with TUI
mothership --tui

# Filter by tags
mothership --only web
mothership --only web,api
mothership --except workers

# Pre-flight check (validate + verify uplinks)
mothership preflight

# View routing chart
mothership chart

# Validate manifest (no network access)
mothership clearance

# Initialize new manifest
mothership init
```

## Base Templates

Reduce duplication with base templates:

```toml
[base.ship]
env = { RAILS_ENV = "production" }
critical = true
tags = ["ruby"]

[base.bay]
critical = true
tags = ["realtime"]

[base.module]
phase = "request"
tags = ["security"]

# Ships inherit from base.ship
[[fleet.web]]
name = "app"
command = "ruby"
tags = ["web"]  # Combined: ["ruby", "web"]

# Bays inherit from base.bay
[[bays.websocket]]
name = "orbitcast"
command = "./orbitcast"
```

## TUI Dashboard

> Requires `--features tui`

The TUI shows real-time fleet status:

- **Overview Tab**: Ship/bay status, group, PID, health, routes
- **Logs Tab**: Per-process stdout/stderr with scroll
- **Modules Tab**: WASM payload status

Controls:
- `Tab` - Switch tabs
- `↑/↓` - Navigate ships
- `PgUp/PgDn` - Scroll logs
- `q` - Quit

## WASM Payloads

> Requires `--features wasm`

Payloads process requests/responses at the proxy layer:

```toml
[[modules]]
name = "auth"
wasm = "./modules/auth.wasm"
routes = ["/admin/.*"]
phase = "request"

[[modules]]
name = "cache"
wasm = "./modules/cache.wasm"
routes = ["/api/.*"]
phase = "response"
config = { ttl = "3600" }
```

Payloads can:
- Block requests (return custom status/body/headers)
- Modify request path/headers
- Modify response headers

Security behavior:
- If configured modules fail to load, startup fails.
- If a module errors during request/response processing, mothership returns `503` (fail-closed).

### Host Functions

WASM modules have access to these host functions:

| Function | Description |
|----------|-------------|
| `get_request()` | Get request JSON length |
| `read_request(ptr, len)` | Read request JSON into buffer |
| `set_action(action)` | Set action: 0=Continue, 1=Block |
| `set_block(status, body_ptr, body_len)` | Block with status and body |
| `set_block_with_headers(status, body_ptr, body_len, headers_ptr, headers_len)` | Block with status, body, and headers (JSON) |
| `log(level, ptr, len)` | Log message (0=debug, 1=warn, 2=info) |

## Static Files & Compression

### Static File Serving

Multiple static directories can be configured. Longest prefix wins, with implicit fallthrough to routes if file not found.

```toml
# Specific assets directory
[[mothership.static_dirs]]
path = "./public/assets"
prefix = "/assets"

# Catch-all for other static files (with fallthrough to routes)
[[mothership.static_dirs]]
path = "./public"
prefix = "/"
bind = "http"  # Optional: limit to specific bind
```

### Response Compression

```toml
[mothership]
compression = true
```

Supports gzip, deflate, and brotli based on `Accept-Encoding`.

### CORS Preflight Cache

Cache CORS preflight (OPTIONS) responses from backends to reduce load:

```toml
[mothership]
cors_cache = true  # Enable with defaults

# Or with custom settings
[mothership.cors_cache]
enabled = true
default_ttl = 3600      # Fallback TTL in seconds (default: 3600)
max_entries = 10000     # Maximum cache entries (default: 10000)
```

Cache key includes: bind name, host, origin, path, `Access-Control-Request-Method`, and `Access-Control-Request-Headers`. TTL is extracted from the backend's `Access-Control-Max-Age` header when present.

```
Browser A → OPTIONS /api/login → Backend → cache + return
Browser B → OPTIONS /api/login → cached ✅ (no backend hit)
Browser C (different origin) → OPTIONS /api/login → Backend → cache + return
```

## Environment Variables

Ships and bays inherit environment variables from mothership:

```bash
MASTER_KEY=secret DATABASE_URL=postgres://... mothership run
```

Process-specific env vars override inherited values:

```toml
[[fleet.web]]
name = "app"
command = "ruby"
env = { RAILS_ENV = "production" }
```

## Prelaunch Jobs

Run setup tasks (migrations, cache warmup, etc.) before any ship or bay starts. All prelaunch jobs must complete successfully or the launch is aborted.

```toml
[[mothership.prelaunch]]
name = "ar-migrate"
command = "rails"
args = ["db:migrate"]

[[mothership.prelaunch]]
name = "memgraph-migrate"
command = "./migrate-memgraph"
depends_on = ["ar-migrate"]  # runs after ar-migrate completes
```

### Prelaunch Configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `name` | string | required | Unique identifier |
| `command` | string | required | Command to execute |
| `args` | string[] | `[]` | Command arguments |
| `env` | table | `{}` | Environment variables |
| `depends_on` | string[] | `[]` | Other prelaunch jobs to wait for |

### Execution Order

1. Uplinks are verified (if configured)
2. Prelaunch jobs run in dependency order
3. Ships and bays launch

If any prelaunch job fails (non-zero exit), the entire launch is aborted.

## Flagship (Multi-Server Coordination)

When deploying multiple Motherships across different servers, the **Flagship** feature ensures only one instance runs prelaunch jobs (migrations, etc.) while others wait.

```
┌─────────────────────────────────────────────────────────────┐
│                         FLEET                               │
│  ┌───────────┐    ┌───────────┐    ┌───────────┐           │
│  │ Server A  │    │ Server B  │    │ Server C  │           │
│  │ Mothership│    │ Mothership│    │ Mothership│           │
│  │ ⭐FLAGSHIP│    │  (escort) │    │  (escort) │           │
│  │           │    │           │    │           │           │
│  │ 1. uplinks│    │ 1. uplinks│    │ 1. uplinks│           │
│  │ 2. migrate│    │ 2. wait...│    │ 2. wait...│           │
│  │ 3. signal │───►│ 3. ready! │    │ 3. ready! │           │
│  │ 4. launch │    │ 4. launch │    │ 4. launch │           │
│  └───────────┘    └───────────┘    └───────────┘           │
└─────────────────────────────────────────────────────────────┘
```

- **Uplinks** are verified on ALL instances (each checks its own connectivity)
- **Prelaunch jobs** run on ONLY ONE instance (the Flagship)
- If the Flagship fails, the **entire deployment aborts**

### Static Election

Explicitly designate the flagship via environment variable or command:

```toml
[mothership.flagship]
enabled = true
election = "static"
static_flagship = "$MS_FLAGSHIP"  # truthy: "true", "1", "yes"
```

Or via command output:

```toml
[mothership.flagship]
enabled = true
election = "static"
static_flagship = { command = "hostname", equals = "server-a" }
```

Examples:
- Kamal: first host runs with `MS_FLAGSHIP=true`
- Heroku: `{ command = "printenv DYNO", equals = "web.1" }`

### PostgreSQL Election

Automatic leader election using advisory locks:

```toml
[mothership.flagship]
enabled = true
election = "postgres"
election_url = "$DATABASE_URL?sslmode=require"
```

The first instance to acquire the lock becomes Flagship. Escorts wait for the ready signal via `LISTEN/NOTIFY`. Requires the `tokio-postgres` feature:

By default, insecure PostgreSQL SSL modes (`disable`, `allow`, `prefer`) are rejected for flagship election. To bypass this (not recommended), set `MOTHERSHIP_ALLOW_INSECURE_POSTGRES_SSLMODE=true`.

```bash
cargo build --features tokio-postgres
```

### Flagship Configuration

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | `false` | Enable flagship coordination |
| `election` | string | `"static"` | Election backend: `"static"` or `"postgres"` |
| `election_url` | string | - | PostgreSQL URL (supports `$ENV` expansion) |
| `static_flagship` | string/table | - | Env var (`"$VAR"`) or command (`{ command, equals }`) |
| `prelaunch_timeout` | u64 | `300` | Seconds escorts wait for flagship |
| `election_timeout` | u64 | `30` | Seconds to acquire lock |

## Uplinks

Verify external dependencies are reachable before launching the fleet. If any uplink fails, mothership aborts startup.

```toml
[[mothership.uplinks]]
url = "postgres://localhost:5432/mydb"
name = "postgres"

[[mothership.uplinks]]
url = "$DATABASE_URL"  # env var expansion
name = "primary-db"
timeout = "10s"  # default: 5s

[[mothership.uplinks]]
url = "redis://cache.internal:6379"
name = "redis"
```

### Pre-flight Command

```bash
mothership preflight
```

Validates the manifest and verifies all uplinks are reachable.

**Note:** `preflight` requires network access to the configured uplinks. Running it on your local machine may fail if uplinks are only accessible from the deployment environment (e.g., internal databases, VPC-only services). Use `clearance` for offline manifest validation.

### Supported Schemes

| Scheme | Default Port | Check |
|--------|--------------|-------|
| `postgres://`, `postgresql://` | 5432 | TCP |
| `mysql://` | 3306 | TCP |
| `redis://` | 6379 | TCP |
| `memgraph://`, `neo4j://` | 7687 | TCP |
| `http://`, `https://` | 80/443 | HTTP GET |
| `tcp://` | required | TCP |

### Deduplication

Uplinks with the same `host:port` are checked only once:

```toml
# These two share the same postgres server - only one TCP check
[[mothership.uplinks]]
url = "postgres://localhost:5432/app_production"
name = "primary"

[[mothership.uplinks]]
url = "postgres://localhost:5432/app_replica"
name = "replica"
```

## Metrics

Enable Prometheus metrics:

```toml
[mothership]
metrics_port = 9090
```

Scrape `http://127.0.0.1:9090/metrics`:

```
mothership_ship_status{ship="app",group="web"} 1
mothership_ship_healthy{ship="app",group="web"} 1
mothership_ship_restarts_total{ship="app",group="web"} 0
mothership_ship_memory_rss_bytes{ship="app",group="web"} 52428800
mothership_ship_memory_virtual_bytes{ship="app",group="web"} 1073741824
mothership_requests_total{route="/api"} 1234
mothership_fleet_ships_total 3
```

Also serves `/health` for liveness probes.

## Rate Limiting

Optional HTTP rate limiting to protect against request floods:

```toml
[mothership.rate_limiting]
global_rps = 10000.0    # Global requests per second (None = unlimited)
per_ip_rpm = 100.0      # Per-IP requests per minute (None = unlimited)
```

When rate limited, clients receive `429 Too Many Requests` with `Retry-After` header.

**Default**: Unlimited (no rate limiting). Enable only if needed.

## Lifecycle States

Mothership tracks its lifecycle through these states:

```
Initializing → Preflight → Electing → Prelaunch → Docking → Launching → Running → Draining → Landed
                   ↓           ↓          ↓          ↓          ↓          ↓
                Failed      Failed     Failed     Failed     Failed    Crashed
```

| State | Description |
|-------|-------------|
| `Initializing` | Loading manifest |
| `Preflight` | Verifying uplinks |
| `Electing` | Flagship coordination |
| `Prelaunch` | Running prelaunch jobs |
| `Docking` | Bays connecting |
| `Launching` | Ships starting |
| `Running` | Normal operation |
| `Mayday` | Distress mode (attempting recovery) |
| `Draining` | Graceful shutdown in progress |
| `Landed` | Clean exit |
| `Failed` | Startup failure |
| `Crashed` | Runtime failure |

The current status is logged on shutdown: `{"message":"Mothership landed","status":"landed"}`

## License

MIT