onwards 0.16.1

A flexible LLM proxy library
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138

# Response Sanitization

Onwards can enforce strict OpenAI API schema compliance for `/v1/chat/completions` responses. This feature:

- **Removes provider-specific fields** from responses
- **Rewrites the model field** to match what the client originally requested
- **Supports both streaming and non-streaming** responses
- **Validates responses** against OpenAI's official API schema
- **Sanitizes error responses** to prevent upstream provider details from leaking to clients

This is useful when proxying to non-OpenAI providers that add custom fields, or when using `onwards_model` to rewrite model names upstream.

> **Note:** For production deployments requiring additional security (request validation, error standardization), consider using [Strict Mode](strict-mode.md) instead, which includes response sanitization plus comprehensive security features.

## Enabling response sanitization

Add `sanitize_response: true` to any target or provider in your configuration.

**Single provider:**

```json
{
  "targets": {
    "gpt-4": {
      "url": "https://api.openai.com",
      "onwards_key": "sk-your-key",
      "onwards_model": "gpt-4-turbo-2024-04-09",
      "sanitize_response": true
    }
  }
}
```

**Pool with multiple providers:**

```json
{
  "targets": {
    "gpt-4": {
      "sanitize_response": true,
      "providers": [
        {
          "url": "https://api1.example.com",
          "onwards_key": "sk-key-1"
        },
        {
          "url": "https://api2.example.com",
          "onwards_key": "sk-key-2"
        }
      ]
    }
  }
}
```

## How it works

When `sanitize_response: true` and a client requests `model: gpt-4`:

1. **Request sent upstream** with `model: gpt-4`
2. **Upstream responds** with custom fields and `model: gpt-4-turbo-2024-04-09`
3. **Onwards sanitizes**:
   - Parses response using OpenAI schema (removes unknown fields)
   - Rewrites `model` field to `gpt-4` (matches original request)
   - Reserializes clean response
4. **Client receives** standard OpenAI response with `model: gpt-4`

## Common use cases

**Third-party providers** (e.g., OpenRouter, Together AI) often add extra fields like `provider`, `native_finish_reason`, `cost`, etc. Sanitization strips these.

**Provider comparison** -- normalize responses from different providers for consistent handling.

**Debugging** -- reduce noise by filtering to only standard OpenAI fields.

## Error sanitization

When `sanitize_response: true`, error responses from upstream providers are also sanitized. This prevents information leakage -- upstream error bodies can contain provider names, internal URLs, and model identifiers that you may not want exposed to clients.

### How it works

Onwards replaces the upstream error body with a generic OpenAI-compatible error, while preserving the original HTTP status code:

- **4xx errors** are replaced with:

```json
{
  "error": {
    "message": "The upstream provider rejected the request.",
    "type": "invalid_request_error",
    "param": null,
    "code": "upstream_error"
  }
}
```

- **5xx errors** (and any other non-2xx status) are replaced with:

```json
{
  "error": {
    "message": "An internal error occurred. Please try again later.",
    "type": "internal_error",
    "param": null,
    "code": "internal_error"
  }
}
```

The original error body is logged at `ERROR` level (up to 64 KB) for debugging, so operators can still investigate upstream failures without exposing details to clients.

### Error format

All Onwards error responses (both sanitized upstream errors and errors generated by Onwards itself) use the OpenAI-compatible `{"error": {...}}` envelope:

```json
{
  "error": {
    "message": "...",
    "type": "...",
    "param": null,
    "code": "..."
  }
}
```

| Field | Description |
|-------|-------------|
| `message` | Human-readable error description |
| `type` | Error category (`invalid_request_error`, `rate_limit_error`, `internal_error`) |
| `param` | The request parameter that caused the error, if applicable |
| `code` | Machine-readable error code |

## Supported endpoints

Currently supports:

- `/v1/chat/completions` (streaming and non-streaming)