Module orchestrator 

Module: Orchestrator

Responsibility

Provides a composable LLM pipeline with circuit breaking, retry, deduplication, and backpressure. Mirrors the public API of tokio-prompt-orchestrator.

Guarantees

• Thread-safe: all types wrap state in Arc<Mutex<_>> or atomics
• Circuit breaker opens after threshold consecutive failures
• RetryPolicy delays grow exponentially and are capped at MAX_RETRY_DELAY
• Deduplicator is deterministic and non-blocking
• BackpressureGuard never exceeds declared hard capacity
• Non-panicking: all operations return Result

NOT Responsible For

• Cross-node circuit breakers (single-process only, unless a distributed backend is provided)
• Persistent deduplication (in-memory, bounded TTL)
• Distributed backpressure

Composing the Primitives

The four primitives are designed to be layered. A typical production setup:

request
  │
  ▼
BackpressureGuard  ← shed if too many in-flight requests
  │
  ▼
Deduplicator       ← return cached result for duplicate keys
  │
  ▼
CircuitBreaker     ← fast-fail if the downstream is unhealthy
  │
  ▼
RetryPolicy        ← retry transient failures with exponential backoff
  │
  ▼
Pipeline           ← transform request/response through named stages

Structs

BackpressureGuard

Tracks in-flight work count and enforces a capacity limit.

CircuitBreaker

Circuit breaker guarding a fallible operation.

Deduplicator

Deduplicates requests by key within a TTL window.

InMemoryCircuitBreakerBackend

In-process circuit breaker backend backed by a Mutex<HashMap>.

Pipeline

A composable pipeline that passes a string through a sequence of named stages.

PipelineResult

Result of executing a pipeline, including per-stage timing.

RetryPolicy

Configurable retry policy with exponential backoff or constant interval.

Stage

A single named stage in the pipeline.

Enums

CircuitState

Tracks failure rates and opens when the threshold is exceeded.

DeduplicationResult

Result of a deduplication check.

RetryKind

Retry mode: exponential backoff or constant interval.

Constants

MAX_RETRY_DELAY

Maximum delay between retries — caps exponential growth.

Traits

AsyncCircuitBreakerBackend

Async counterpart of CircuitBreakerBackend for distributed backends.

CircuitBreakerBackend

Backend for circuit breaker state storage.