Module response_optimizer 

Response Optimizer (P9 / DIM 2 — Output-Optimierung).

Reduces output tokens without semantic loss through two mechanisms:

1. Response Cache — identical user queries within a session get the cached response instead of a full LLM round-trip. Saves 100% of output tokens on cache hits.

2. Response Dedup — detects when the model repeats substantially similar answers within a conversation and signals this to the client (future: truncate/summarize repeated content).

These complement the existing mechanisms:

• verbosity.rs — wire-level “be concise” steer (reduces verbosity ~33%)
• output_savings.rs — A/B measurement of output reduction
• effort_routing.rs — thinking budget control

Opt-in only (proxy.response_cache = true). Off by default.

Cache design

• Key: BLAKE3 hash of (model + last N user messages + system prompt)
• Value: the complete streamed response (reassembled)
• TTL: configurable, default 5 minutes (short — LLM answers can evolve)
• Capacity: bounded LRU, default 64 entries per session
• Scope: per-session (not cross-session — avoids stale context leaks)

Dedup design

• Tracks BLAKE3 fingerprints of recent responses (last 16)
• A response whose first 200 chars match a recent fingerprint is flagged
• Flagging is observability-only in v1 (no truncation)

Determinism

Cache hits are deterministic: same key always returns the same value. Cache misses are non-deterministic (LLM output varies), but the decision to serve from cache vs. forward is deterministic given the cache state.

Structs

DedupTracker

Response deduplication tracker.

OptimizationDecision

An optimization decision record.

OptimizerStats

Optimizer statistics for observability.

ResponseCache

The response cache — bounded LRU with TTL eviction.

ResponseOptimizerConfig

Configuration for the response optimizer.

SessionOptimizer

Per-session optimizer state. Each session/conversation gets its own instance.

Enums

OptimizationSource

What triggered the optimization.

Functions

compute_cache_key

Compute a cache key from the request components that determine the response. Uses a fast non-cryptographic hash (FxHash-style) for performance.

fingerprint_response

Compute a fingerprint of a response for dedup detection. Uses the first 200 chars to catch repeated preambles/patterns.

get_or_create

Get or create the optimizer for a session.

global_stats

Global statistics across all sessions.

optimize_response

Apply the proxy optimizer to an OCLA response decision.

remove_session

Remove a session’s optimizer (cleanup on session end).