# Agent Trace Infra HTTP API
Standalone **Trace Infra Server** — diagnostic trace event storage for Agent Runtime.
| Binary | `agent-trace-infra` |
| Default bind | `127.0.0.1:5140` |
| Audience / service name | `agent-trace-infra` |
| OpenAPI-style prefix | `/internal/v1/trace-infra` |
This server provides **idempotent, chronologically ordered** trace event storage. Trace events are diagnostic records of tool calls, guard-rail evaluations, and lifecycle transitions inside a single run. Unlike conversation messages, trace events are **never replayed into a model prompt**.
---
## Authentication
| **Open** | `AGENT_TRACE_INFRA_SERVICE_TOKEN` unset / empty | All routes accept requests without `Authorization` |
| **Token** | env set to a non-empty string | Internal routes require `Authorization: Bearer <token>` |
- `GET /health` is always unauthenticated.
- Wrong / missing token → `401` / `403` with body `{ "status": <code>, "message": "..." }`.
```bash
export AGENT_TRACE_INFRA_SERVICE_TOKEN='<development-token>'
curl -H "Authorization: Bearer <development-token>" http://127.0.0.1:5140/internal/v1/trace-infra/events/batch \
-H 'content-type: application/json' \
-d '{"events":[{"id":"e1","scope":{"conversationId":"c1","runId":"r1"},"eventType":"tool_call","payload":{}}]}'
```
---
## Error shape
All non-2xx JSON errors:
```json
{ "status": 400, "message": "human readable reason" }
```
| 400 | Validation failure: empty id, empty conversationId, etc. |
| 401 | Missing bearer when token mode is on |
| 403 | Bearer present but wrong |
---
## `GET /health`
**Auth:** none
**Response `200`:**
```json
{ "status": "ok", "service": "agent-trace-infra" }
```
---
## `POST /internal/v1/trace-infra/events/batch`
Append trace events. Idempotent: re-sending an event with the same `id` within the same partition is a no-op (counted in `skipped`).
**Auth:** token mode if configured
### Request body
```json
{
"events": [
{
"id": "e1",
"scope": { "conversationId": "c1", "runId": "r1" },
"eventType": "tool_call",
"payload": { "tool": "search", "args": {} },
"sequence": 0,
"createdAtMs": 0
}
]
}
```
| `events` | `[RuntimeEventRecord]` | yes | |
| `events[].id` | string | yes | Used for idempotent deduplication |
| `events[].scope.conversationId` | string | yes | Partition key (part 1) |
| `events[].scope.runId` | string | no | Partition key (part 2); groups events from one turn |
| `events[].eventType` | string | yes | Coarse kind: `tool_call`, `guard_rail`, `turn_started`, `turn_finished`, etc. |
| `events[].payload` | JSON value | no | Free-form payload; auto-redacted on store |
| `events[].metadata` | JSON value | no | Free-form metadata; auto-redacted on store |
| `events[].sequence` | i64 | no | `0` = auto-assign (monotonic per partition) |
| `events[].createdAtMs` | i64 | no | `0` = auto-fill to current epoch millis |
### Response `200`
```json
{ "appended": 1, "skipped": 0 }
```
| `appended` | usize | Number of events newly stored |
| `skipped` | usize | Number of events omitted because their `id` already existed in the same partition |
### curl
```bash
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/events/batch \
-H 'content-type: application/json' \
-d '{"events":[{"id":"e1","scope":{"conversationId":"c1","runId":"r1"},"eventType":"tool_call","payload":{"name":"execute_cmd"}}]}'
```
---
## `POST /internal/v1/trace-infra/events/search`
Search for events in a conversation/run partition or browse the newest events for a Runtime.
Returns events in chronological order, oldest first.
**Auth:** token mode if configured
### Request body
```json
{
"scope": { "conversationId": "c1", "runId": "r1" },
"eventType": "tool_call",
"limit": 100
}
```
| `scope` | `TraceScope` | yes | `conversationId` (+ optional `runId`) or `runtimeId` |
| `eventType` | string | no | When set, only events with this exact type are returned |
| `limit` | usize | no | Default `100`, clamped to `1000` |
### Response `200`
```json
{
"items": [
{
"id": "e1",
"scope": { "conversationId": "c1", "runId": "r1" },
"eventType": "tool_call",
"payload": { "name": "execute_cmd" },
"sequence": 1,
"createdAtMs": 1700000000000
}
]
}
```
Records are in **chronological order, oldest first**. The `limit` applies to the tail of the filtered list: you get the last `limit` records.
Control-plane callers may instead submit
`{"scope":{"conversationId":"","runtimeId":"rti_123"},"limit":500}`. Runtime lookup uses the
persisted `(runtime_id, created_at_ms, sequence)` index; events produced by older Runtime SDKs are
indexed from allow-listed `metadata.runtimeId` during append.
### curl
```bash
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/events/search \
-H 'content-type: application/json' \
-d '{"scope":{"conversationId":"c1","runId":"r1"},"limit":100}'
```
---
## Guarantees
### Ordering
All reads return events in **chronological order, oldest first**, sorted by `(created_at_ms, sequence)`. Ties break by `sequence`.
### Idempotency
Append is idempotent by `id`. If an event with the same `id` already exists in the same partition, the new event is silently dropped and counted in `skipped`. This makes retry after a network timeout safe: you will never get duplicate trace events.
### Per-run partitioning
Events are partitioned by `(conversation_id, run_id)`. Events from one run are never mixed with another run's events, even when they share the same conversation. This makes it safe to reconstruct a single execution from its trace without guessing from timestamps.
### Server-side redaction
Every event's `payload` and `metadata` are redacted before storage. Fields whose names look like credentials (`api_key`, `token`, `authorization`, `secret`, `password`, etc.) are replaced with `<redacted>`. This is done server-side because trace data is the most likely to be exported to a log aggregator and retained — a leaked credential there outlives the request that leaked it.
### Bounded retention (per-partition)
Trace is high-volume: a single turn can produce dozens of events. The store keeps at most 10,000 events per partition (configurable via `TRACE_INFRA_MAX_EVENTS_PER_RUN`), evicting the oldest entries when the cap is exceeded. This prevents the store from becoming a slow memory leak.
### Trace events are never replayed into a model prompt
Unlike conversation messages, trace events are diagnostic records. They are never replayed into a model prompt, so losing one is acceptable. This is the fundamental design distinction that justifies a separate service.
---
## Page-size limits
| `DEFAULT_PAGE_SIZE` | 100 |
| `MAX_PAGE_SIZE` | 1000 |
An unbounded read is a denial-of-service vector, so every page-size parameter is capped at `MAX_PAGE_SIZE`. The default is `DEFAULT_PAGE_SIZE` when the request omits the field.
---
## Langfuse Metrics API
Token-usage, cost, latency, and error-rate metrics sourced from a Langfuse ClickHouse database. All endpoints are `POST` and accept the same `MetricsQuery` request body.
### Request body
```json
{
"projectIds": ["proj_1", "proj_2"],
"days": 30,
"dataSource": "auto",
"v4CutoverDate": "2026-09-01 00:00:00"
}
```
| `projectIds` | `string[]` | null (all) | Project ID filter |
| `days` | `number` | `30` | Look-back window |
| `dataSource` | `"v3" \| "v4" \| "auto"` | `"auto"` | ClickHouse table to query |
| `v4CutoverDate` | `string` | `"2026-09-01 00:00:00"` | Date boundary for v3/v4 auto-routing |
### `POST /internal/v1/trace-infra/metrics/token-usage`
Daily token consumption timeseries, grouped by project.
**Response `200`:**
```json
{
"proj_1": {
"projectId": "proj_1",
"points": [
{
"day": "2026-07-12 00:00:00",
"totalInputTokens": 50000,
"totalOutputTokens": 30000,
"totalTokens": 80000,
"observationCount": 120,
"generationCount": 100,
"totalCost": 0.15
}
]
}
}
```
### `POST /internal/v1/trace-infra/metrics/model-breakdown`
Per-model token and cost breakdown across all projects.
**Response `200`:**
```json
[
{
"projectId": "proj_1",
"modelName": "gpt-4",
"observationCount": 500,
"totalInputTokens": 100000,
"totalOutputTokens": 50000,
"totalTokens": 150000,
"totalCost": 3.0,
"pricingTier": "Standard"
}
]
```
### `POST /internal/v1/trace-infra/metrics/project-summary`
Aggregated statistics per project.
**Response `200`:**
```json
[
{
"projectId": "proj_1",
"totalTokens": 150000,
"totalInputTokens": 100000,
"totalOutputTokens": 50000,
"observationCount": 600,
"generationCount": 500,
"totalCost": 3.0,
"uniqueModels": 3,
"uniqueTraces": 25
}
]
```
### `POST /internal/v1/trace-infra/metrics/latency`
Latency percentiles (p50/p95/p99) and average TTFT, grouped by project, day, and model.
**Response `200`:**
```json
{
"proj_1": {
"projectId": "proj_1",
"points": [
{
"day": "2026-07-12 00:00:00",
"projectId": "proj_1",
"modelName": "gpt-4",
"observationCount": 50,
"avgDurationSeconds": 2.5,
"p50DurationSeconds": 1.8,
"p95DurationSeconds": 5.2,
"p99DurationSeconds": 8.1,
"avgTtftSeconds": 0.3
}
]
}
}
```
### `POST /internal/v1/trace-infra/metrics/error-rate`
Error rate per project, day, and model.
**Response `200`:**
```json
{
"proj_1": {
"projectId": "proj_1",
"points": [
{
"day": "2026-07-12 00:00:00",
"projectId": "proj_1",
"totalObservations": 100,
"errorCount": 3,
"errorRate": 0.03,
"modelName": "gpt-4"
}
]
}
}
```
### `POST /internal/v1/trace-infra/metrics/cache-hit-rate`
Cache hit rate per model (P0). Only returns models with `cached_tokens > 0`.
**Response `200`:**
```json
{
"items": [
{
"projectId": "proj_1",
"modelName": "deepseek-chat",
"totalInputTokens": 100000,
"cachedTokens": 30000,
"cacheHitPct": 30.0
}
]
}
```
### `POST /internal/v1/trace-infra/metrics/cost-breakdown`
Input/output/total cost breakdown per model (P1).
**Response `200`:**
```json
{
"items": [
{
"projectId": "proj_1",
"modelName": "gpt-4",
"totalInputCost": 2.0,
"totalOutputCost": 1.5,
"totalCost": 3.5,
"observationCount": 500
}
]
}
```
### `POST /internal/v1/trace-infra/metrics/model-trend`
Daily model usage trends, grouped by project (P1).
**Response `200`:**
```json
{
"proj_1": {
"projectId": "proj_1",
"points": [
{
"day": "2026-07-12 00:00:00",
"projectId": "proj_1",
"modelName": "gpt-4",
"callCount": 50,
"totalTokens": 80000,
"totalCost": 1.2
}
]
}
}
```
### `POST /internal/v1/trace-infra/metrics/token-distribution`
Token distribution statistics (avg/p50/p95/p99/max) by model (P2).
**Response `200`:**
```json
{
"items": [
{
"projectId": "proj_1",
"modelName": "gpt-4",
"observationCount": 500,
"avgTokens": 300.0,
"p50Tokens": 250.0,
"p95Tokens": 800.0,
"p99Tokens": 1200.0,
"maxTokens": 2000
}
]
}
```
### `POST /internal/v1/trace-infra/metrics/activity`
Daily activity metrics (active traces, observation count) per project (P2).
**Response `200`:**
```json
{
"proj_1": {
"projectId": "proj_1",
"points": [
{
"day": "2026-07-12 00:00:00",
"projectId": "proj_1",
"activeTraces": 25,
"observationCount": 500
}
]
}
}
```
### `POST /internal/v1/trace-infra/metrics/pricing-tier`
Pricing tier analysis per model (P2).
**Response `200`:**
```json
{
"items": [
{
"projectId": "proj_1",
"pricingTier": "Standard",
"modelName": "gpt-4",
"observationCount": 500,
"totalTokens": 150000,
"totalCost": 3.0
}
]
}
```
---
## Environment variables
| `AGENT_TRACE_INFRA_SERVICE_TOKEN` | Static bearer token for auth; unset/empty = open mode |
| `TRACE_INFRA_MAX_EVENTS_PER_RUN` | Per-partition retention cap (default 10,000) |
| `CLICKHOUSE_HOST` | **Required** for metrics; ClickHouse server address |
| `CLICKHOUSE_PORT` | ClickHouse HTTP port (default `8123`; native `9000` is mapped to `8123`) |
| `CLICKHOUSE_DATABASE` | ClickHouse database name (default `fx_langfuse`) |
| `CLICKHOUSE_USER` | ClickHouse user (default `default`) |
| `CLICKHOUSE_PASSWORD` | ClickHouse password (default empty) |
| `CLICKHOUSE_TIMEOUT_SECS` | HTTP query timeout in seconds (default `30`, max `300`) |
| `LANGFUSE_BASE_URL` | Langfuse server URL used by the project admin client |
| `LANGFUSE_PUBLIC_KEY` | Organization-level Langfuse public key |
| `LANGFUSE_SECRET_KEY` | Organization-level Langfuse secret key |
| `LANGFUSE_CREDENTIALS_ENCRYPTION_KEY` | Optional stable key for project credentials at rest; falls back to `LANGFUSE_SECRET_KEY` |
The three `LANGFUSE_*` connection values are all-or-none bootstrap. Partial
configuration fails at startup. After the first successful Admin
`PUT /admin/api/langfuse-config` or `PUT /admin/api/clickhouse-config`, the
matching environment variables are no longer the connection source. Durable
project relationships require `--db-url` / `AGENT_TRACE_DB_URL`; the local
in-memory mode is intended for development only.
---
## Langfuse project administration
The Gateway exposes these Trace-owned endpoints under the unified admin namespace:
| `GET /admin/api/langfuse-config` | `GET /internal/v1/trace-infra/langfuse-config` | `infra-admin:read` |
| `PUT /admin/api/langfuse-config` | `PUT /internal/v1/trace-infra/langfuse-config` | `infra-admin:write` |
| `GET /admin/api/clickhouse-config` | `GET /internal/v1/trace-infra/clickhouse-config` | `infra-admin:read` |
| `PUT /admin/api/clickhouse-config` | `PUT /internal/v1/trace-infra/clickhouse-config` | `infra-admin:write` |
| `POST /admin/api/langfuse-projects` | `POST /internal/v1/trace-infra/langfuse-projects` | `infra-admin:write` |
| `GET /admin/api/langfuse-projects` | `GET /internal/v1/trace-infra/langfuse-projects` | `infra-admin:read` |
| `GET /admin/api/langfuse-projects/{projectId}` | `GET /internal/v1/trace-infra/langfuse-projects/{projectId}` | `infra-admin:read` |
Create requires `Idempotency-Key` and at least one relation:
```json
{
"name": "Weather Agent",
"ownerEmail": "owner@example.com",
"relations": {
"agentIds": ["agt_weather"],
"runtimeIds": ["runtime-weather"],
"userIds": ["user-weather"]
}
}
```
Trace calls Langfuse `POST /api/public/projects`, then
`POST /api/public/projects/{projectId}/apiKeys`. A successful create returns `201` with
`{ "project": ..., "secretKey": "..." }`. Replaying the same idempotency key and request returns
the same credentials; reusing the key with a different request returns `409`.
List supports `relatedAgentId`, `relatedRuntimeId`, `relatedUserId`, `cursor`, and `limit` (maximum
200). List/get responses never include the project secret.
### Runtime binding
`GET /v1/trace/runtime-bindings/{runtimeId}` (Trace upstream
`/internal/v1/trace-infra/runtime-bindings/{runtimeId}`) requires `trace:read` and
returns the Langfuse project bound to the authenticated `(tenant_id, project_id, runtimeId)`
tuple. Missing bindings return `404`. Responses never include the project secret.
`POST /v1/trace/runtime-bindings:resolve` (Trace upstream
`/internal/v1/trace-infra/runtime-bindings:resolve`) requires `trace:read` and accepts up to 200
runtime ids. It returns existing bindings plus `unboundRuntimeIds` in first-seen request order, so
inventory and usage screens can resolve sparse bindings without using expected `404` responses as
control flow. Duplicate runtime ids are resolved once.
`POST /v1/trace/runtime-bindings/{runtimeId}/initialize` (Trace upstream
`/internal/v1/trace-infra/runtime-bindings/{runtimeId}/initialize`) requires
`Idempotency-Key` and `trace:write`. The binding key is the authenticated
`(tenant_id, project_id, runtimeId)` tuple, never a body-supplied scope. Replays for the same tuple
return its existing Langfuse project even when a caller regenerates the idempotency key. Raw
idempotency keys are namespaced by tenant/project/runtime before project allocation, so a key used in one
scope cannot replay credentials from another scope.
The binding table stores only the Langfuse project id for that tuple. There is no PUT override of
per-runtime Langfuse credentials.
Runtime workload credentials may initialize only their own runtime identity (or a runtime explicitly
present in credential relations). When `agentId` is supplied with runtime authentication, that agent
must also be present in credential relations. Gateway user calls enforce any resource relations
available in their claims. The authenticated principal email overrides request `ownerEmail` when
available; callers without an email claim retain the validated request value or the runtime fallback.
The compatible migration creates `trace_runtime_langfuse_bindings` without assigning legacy global
projects to a tenant. Existing projects remain visible to admin audit, while the first scoped
initialize creates a binding whose ownership is unambiguous.
---
## Curl cookbook (local open mode)
```bash
# Health
curl -s http://127.0.0.1:5140/health
# Append an event
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/events/batch \
-H 'content-type: application/json' \
-d '{"events":[{"id":"e1","scope":{"conversationId":"c1","runId":"r1"},"eventType":"tool_call","payload":{"name":"ls"}}]}'
# Append a second event (different run)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/events/batch \
-H 'content-type: application/json' \
-d '{"events":[{"id":"e2","scope":{"conversationId":"c1","runId":"r2"},"eventType":"observation","payload":{"result":"ok"}}]}'
# Search events in a run (chronological, oldest first)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/events/search \
-H 'content-type: application/json' \
-d '{"scope":{"conversationId":"c1","runId":"r1"},"limit":100}'
# Filter by event type
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/events/search \
-H 'content-type: application/json' \
-d '{"scope":{"conversationId":"c1","runId":"r1"},"eventType":"tool_call"}'
# Token usage (last 30 days, all projects)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/token-usage \
-H 'content-type: application/json' \
-d '{"days":30}'
# Model breakdown (specific projects, v4 data source)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/model-breakdown \
-H 'content-type: application/json' \
-d '{"projectIds":["proj_1","proj_2"],"days":7,"dataSource":"v4"}'
# Cache hit rate (P0)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/cache-hit-rate \
-H 'content-type: application/json' \
-d '{"days":30}'
# Cost breakdown (P1)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/cost-breakdown \
-H 'content-type: application/json' \
-d '{"days":30}'
# Model trends (P1)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/model-trend \
-H 'content-type: application/json' \
-d '{"days":30}'
# Token distribution (P2)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/token-distribution \
-H 'content-type: application/json' \
-d '{"days":30}'
# Activity (P2)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/activity \
-H 'content-type: application/json' \
-d '{"days":30}'
# Pricing tier (P2)
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/pricing-tier \
-H 'content-type: application/json' \
-d '{"days":30}'
# Latency percentiles
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/latency \
-H 'content-type: application/json' \
-d '{"days":7}'
# Error rate
curl -s -X POST http://127.0.0.1:5140/internal/v1/trace-infra/metrics/error-rate \
-H 'content-type: application/json' \
-d '{"days":1}'
```
---
## Related crates
| `agent-trace-contract` | Types + path constants + this API surface |
| `agent-trace-runtime` | Trace store + ClickHouse metrics client implementation |
| `agent-trace-server` | Axum host + optional static token auth |