---
title: Health Checks
description: Automatic background health monitoring that removes and restores backends without restarting.
---
Health checks are available in **config-driven mode** only. Each `[[upstream]]` — and, for WebSocket backends, each `[[ws_proxy]]` — that includes a `[*.health_check]` section gets a dedicated daemon thread (`health-{name}`) that probes every backend on a regular interval and updates the live backend list without any restart. Both use the exact same checker (`start_health_checker`), so everything on this page applies to `[ws_proxy.health_check]` too, unless noted otherwise.
## Configuration
```toml
[[upstream]]
name = "api"
backends = ["api-1:3000", "api-2:3000", "api-3:3000"]
[upstream.health_check]
path = "/healthz" # GET path (default: "/health")
interval_secs = 15 # probe interval in seconds (default: 30)
timeout_ms = 3000 # connect + read timeout per probe (default: 5000)
healthy_threshold = 2 # consecutive successes to restore (default: 2)
unhealthy_threshold = 3 # consecutive failures to remove (default: 3)
```
Same fields for a WebSocket proxy:
```toml
[[ws_proxy]]
name = "chat"
listen = "0.0.0.0:9000"
backends = ["wss://chat-a:8443", "wss://chat-b:8443"]
[ws_proxy.health_check]
path = "/healthz"
interval_secs = 10
timeout_ms = 2000
healthy_threshold = 2
unhealthy_threshold = 3
```
The probe is a plain HTTP `GET` against the backend's `host:port` — not a real WebSocket handshake — connected over TLS when the backend uses `wss://`. This is deliberate: nginx and Traefik health-check WebSocket upstreams the same way, because a WS backend is almost always a regular HTTP server with an upgrade route, not a WS-only listener, so a plain HTTP endpoint is enough signal that the process and its network stack are up.
## How it works
### Startup state
All backends start as **live**. The health checker assumes backends are healthy until proven otherwise.
### Probe request
Every `interval_secs` seconds the checker sends a minimal HTTP/1.1 request to each backend:
```
GET /healthz HTTP/1.1
Host: api-1
Connection: close
```
Both the TCP connect and the response read are bounded by `timeout_ms`. A backend is considered healthy if it replies with a `2xx` status code (the checker reads only the first 16 bytes of the response — just enough for `HTTP/1.1 2`).
### Failure tracking
Per-backend counters track consecutive successes and failures independently:
```
backend api-2:
consecutive failures = 1 → still live
consecutive failures = 2 → still live
consecutive failures = 3 → REMOVED from live list (unhealthy_threshold reached)
backend api-2 later:
consecutive successes = 1 → still dead
consecutive successes = 2 → RESTORED to live list (healthy_threshold reached)
```
The counters reset on state transition: a success resets the failure counter to `0`, and vice versa.
### Live list update
After probing all backends, the checker atomically replaces the shared live list:
```rust
Arc<RwLock<Vec<String>>> // written by health checker; read by DynamicProxy
```
`DynamicProxy` acquires a read lock on every request, which is concurrent-safe. The health checker acquires a write lock once per probe round, to publish the new list.
### Log output
State changes are logged to stderr:
```
[health] upstream=api backend=api-2:3000 removed (3x fail)
[health] upstream=api backend=api-2:3000 restored (2x ok)
```
## Passive health checks
Active probes find a dead backend only on the next interval. `[upstream.circuit_breaker]` also takes a backend out when the requests it serves fail (no response, `502`/`503`/`504` — `failure_threshold` in a row) and lets a probe request through after `recovery_secs`. The two combine: a backend is in rotation only while it passes its probes and its breaker isn't open. See [Weights, stickiness, retries and the circuit breaker](/proxy/config-driven/#weights-stickiness-retries-and-the-circuit-breaker).
## All backends unhealthy
If all backends fail their health checks, the live list becomes empty. `DynamicProxy` returns `502 Bad Gateway` for every HTTP/gRPC request until at least one backend recovers (`503` with `Retry-After` when the circuit breaker has taken them all out); a `[[ws_proxy]]` returns `503 Service Unavailable` for every new WebSocket upgrade attempt instead (the connection never gets far enough to be a "gateway" in the HTTP sense — it's rejected before any backend is contacted).
:::note[No health checks in library mode — except WsProxy]
`ReverseProxy` does not have built-in health checks. Use `BackendPool` with a `File` or `Dns` source (see [Service Discovery](/proxy/service-discovery/)) and `CircuitBreaker` (see [Circuit Breaker](/proxy/circuit-breaker/)) to replicate the behaviour programmatically.
`WsProxy` is the exception: `WsProxy::with_live_backends(all_backends, live)` takes an `Arc<RwLock<Vec<String>>>` you update from a health-check thread you write yourself — useful if you want a probe other than a plain HTTP `GET` (e.g. a real WebSocket handshake). See [WebSocket Proxy](/proxy/websocket-proxy/#health-checks).
:::
## Implementation reference
The health checker lives in `src/proxy_config/health.rs`:
- `start_health_checker(upstream_name, backends, live, config, stop, trust)` — spawns the daemon thread. It exits when `stop` is set (a `SIGHUP` reload sets it for the replaced configuration's checkers); `trust` is the upstream's `tls_ca_file` roots, if any, so probes to `https://` backends verify against the same CA as requests.
- `check_backend(backend, path, timeout, trust)` — sends a single probe, returns `true` on `2xx`.
- `parse_backend_url(backend)` — strips the `https://`/`http://`/`h2://`/`wss://`/`ws://` prefix (or none) and returns `(host, port, tls)`.
`WsProxy`'s side of this lives in `src/ws_proxy/mod.rs`:
- `WsProxy::with_live_backends(all_backends, live)` — the constructor `[ws_proxy.health_check]` uses internally, also public for library use.
- `pick_backend()` reads the current `live` list on every new connection and returns `None` (→ `503`) if it's empty.