Struct Context 

pub struct Context<CTXEXT> {
    pub ext: Arc<CTXEXT>,
    pub cost_multiplier: Decimal,
    pub ensemble_cache: Arc<DashMap<String, Shared<Receiver<Result<Option<(Ensemble, u64)>, ResponseError>>>>>,
    pub ensemble_llm_cache: Arc<DashMap<String, Shared<Receiver<Result<Option<(EnsembleLlm, u64)>, ResponseError>>>>>,
}

Per-request context containing user-specific state and deduplication caches.

The context is generic over CTXEXT, allowing custom extensions for different deployment scenarios (e.g., different BYOK providers).

Caches

The caches deduplicate concurrent fetches for the same resource within a request. When multiple parts of a request need the same ensemble or ensemble LLM, only one fetch is performed and the result is shared.

Fields

ext: Arc<CTXEXT>

Custom context extension (e.g., for BYOK keys).

cost_multiplier: Decimal

Multiplier applied to costs for this request.

ensemble_cache: Arc<DashMap<String, Shared<Receiver<Result<Option<(Ensemble, u64)>, ResponseError>>>>>

Cache for ensemble fetches, keyed by ensemble ID.

ensemble_llm_cache: Arc<DashMap<String, Shared<Receiver<Result<Option<(EnsembleLlm, u64)>, ResponseError>>>>>

Cache for ensemble LLM fetches, keyed by ensemble LLM ID.

Implementations

impl<CTXEXT> Context<CTXEXT>

pub fn new(ext: Arc<CTXEXT>, cost_multiplier: Decimal) -> Self

Creates a new context with the given extension and cost multiplier.

Trait Implementations

impl<CTXEXT> Clone for Context<CTXEXT>

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<CTXEXT: Debug> Debug for Context<CTXEXT>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.