Struct CachingFetcher 

pub struct CachingFetcher<CTXEXT, FENSLLM> {
    pub inner: Arc<FENSLLM>,
    /* private fields */
}

Wraps an Ensemble LLM fetcher with per-request deduplication caching.

When multiple parts of a request need the same Ensemble LLM, this fetcher ensures only one actual fetch is performed. Subsequent requests for the same Ensemble LLM ID within the same request context share the result.

Fields

inner: Arc<FENSLLM>

The underlying fetcher to delegate to on cache miss.

Implementations

impl<CTXEXT, FENSLLM> CachingFetcher<CTXEXT, FENSLLM>

pub fn new(inner: Arc<FENSLLM>) -> Self

Creates a new caching fetcher wrapping the given inner fetcher.

impl<CTXEXT, FENSLLM> CachingFetcher<CTXEXT, FENSLLM>

where CTXEXT: Send + Sync + 'static, FENSLLM: Fetcher<CTXEXT> + Send + Sync + 'static,

pub fn spawn_fetches<'id>( &self, ctx: Context<CTXEXT>, ids: impl Iterator<Item = &'id str>, )

Spawns concurrent fetch tasks for multiple Ensemble LLM IDs.

This allows pre-warming the cache when the set of required IDs is known ahead of time, reducing latency by parallelizing the fetches.

pub async fn fetch( &self, ctx: Context<CTXEXT>, id: &str, ) -> Result<Option<(EnsembleLlm, u64)>, ResponseError>

Fetches an Ensemble LLM, using the request-scoped cache for deduplication.

If another fetch for the same ID is already in progress within this request context, waits for and shares that result instead of fetching again.

Trait Implementations

impl<CTXEXT: Clone, FENSLLM: Clone> Clone for CachingFetcher<CTXEXT, FENSLLM>

fn clone(&self) -> CachingFetcher<CTXEXT, FENSLLM>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<CTXEXT: Debug, FENSLLM: Debug> Debug for CachingFetcher<CTXEXT, FENSLLM>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.