zeph 0.21.2

Lightweight AI agent with hybrid inference, skills-first architecture, and multi-channel I/O
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143

# Prometheus Monitoring

Zeph can expose a `/metrics` endpoint in [OpenMetrics](https://openmetrics.io/) format that
Prometheus can scrape. A pre-built Grafana dashboard is included for instant visualization.

## Prerequisites

- Zeph built with the `prometheus` feature (included in `server` and `full` feature sets)
- Docker (for the bundled Prometheus + Grafana stack)

## Enable the Metrics Endpoint

In your `config.toml`:

```toml
[gateway]
enabled = true
port = 8090

[metrics]
enabled = true
path = "/metrics"
sync_interval_secs = 5
```

The `prometheus` feature implies `gateway`, so you only need to enable the gateway once.

Verify the endpoint is live:

```bash
curl http://localhost:8090/metrics
```

You should see OpenMetrics text output ending with `# EOF`.

## Start the Monitoring Stack

```bash
docker compose -f docker/docker-compose.metrics.yml up
```

This starts:
- **Prometheus** on `http://localhost:9090` — scrapes Zeph every 10 seconds
- **Grafana** on `http://localhost:3000` — pre-configured with the Zeph dashboard

Both services include health checks; Grafana waits until Prometheus passes its health check before starting.

Open Grafana at `http://localhost:3000`. No login is required in the default configuration
(anonymous viewer access is enabled). The **Zeph Overview** dashboard is available under
**Dashboards → Zeph**.

### Custom Metrics Host

If Zeph listens on a different host or port, edit `docker/prometheus/prometheus.yml` and update
the `static_configs.targets` value. Prometheus does not support environment variable substitution
in its config file.

```bash
# 1. Edit docker/prometheus/prometheus.yml: change targets to ["192.168.1.10:9000"]
# 2. Start the stack:
docker compose -f docker/docker-compose.metrics.yml up
```

### Linux Networking

`host.docker.internal` resolves automatically on Docker Desktop (macOS/Windows) and on
Docker Engine >= 20.10 with the `extra_hosts: host.docker.internal:host-gateway` entry already
set in `docker-compose.metrics.yml`. On older Linux setups, set `network_mode: host` on the
`prometheus` service in the compose file instead.

## Running Alongside the Docker Stack

If Zeph is running inside Docker (e.g. `docker-compose.yml`), add the metrics overlay:

```bash
docker compose -f docker/docker-compose.yml -f docker/docker-compose.metrics.yml up
```

Then edit `docker/prometheus/prometheus.yml` to scrape the Zeph container instead of the host:

```yaml
scrape_configs:
  - job_name: "zeph"
    static_configs:
      - targets: ["zeph:8090"]   # Docker service name
```

## Dashboard Panels

The **Zeph Overview** dashboard includes these panel rows:

| Row | Metrics |
|-----|---------|
| LLM Performance | Token rate, API call rate, last-call latency, context tokens |
| LLM Latency Histograms | p50/p95/p99 for LLM calls, turns, and tool executions |
| Agent Turn Phases | Last/average/max duration per phase (prepare_context, llm_chat, tool_exec, persist) |
| Memory & Context | Message count, embedding rate, compaction rate, Qdrant status |
| Tools & Cache | Cache hit/miss rate, tool output prune rate |
| Security | Injection flags, exfiltration blocks, quarantine invocations, rate-limit trips |
| System | Uptime, skills loaded, MCP server status, background task counts, orchestration rates |

## Custom Prometheus Configuration

If you already have a Prometheus instance, add Zeph as a scrape target:

```yaml
scrape_configs:
  - job_name: "zeph"
    static_configs:
      - targets: ["<zeph-host>:8090"]
    metrics_path: "/metrics"
    scrape_interval: 10s
```

Replace `<zeph-host>` with the hostname or IP where Zeph is running.

## Change the Admin Password

Set `GRAFANA_ADMIN_PASSWORD` before starting the stack:

```bash
GRAFANA_ADMIN_PASSWORD=mysecret docker compose -f docker/docker-compose.metrics.yml up
```

## Troubleshooting

**`curl http://localhost:8090/metrics` returns connection refused**

Check that both `[gateway] enabled = true` and `[metrics] enabled = true` are set in your config.
The gateway binds to `0.0.0.0:8090` by default.

**Prometheus shows `zeph` target as DOWN**

On Linux, `host.docker.internal` requires Docker Engine 20.10+ with `--add-host`.
If your setup doesn't support it, switch to `network_mode: host` in
`docker/docker-compose.metrics.yml` for the `prometheus` service, or use the container name
when Zeph runs inside Docker.

**No data in Grafana**

Confirm Prometheus can reach the metrics endpoint: open `http://localhost:9090/targets` and
check that `zeph` is in state UP. If the target is down, verify the `targets` in
`docker/prometheus/prometheus.yml` matches where Zeph is listening.