Struct CachedInference 

pub struct CachedInference {
    pub cache: SemanticCache,
}

Middleware wrapper that checks the semantic cache before running inference.

use oxibonsai_runtime::semantic_cache::{CachedInference, SemanticCacheConfig};

let ci = CachedInference::new(SemanticCacheConfig::default());

// First call: cache miss — closure runs.
let (resp, hit) = ci.run_or_cache(
    "What is the capital of France?",
    || "Paris is the capital of France.".to_string(),
);
assert!(!hit);
assert_eq!(resp, "Paris is the capital of France.");

Fields

cache: SemanticCache

The underlying semantic cache. Exposed so callers can inspect stats.

Implementations

impl CachedInference

pub fn new(config: SemanticCacheConfig) -> Self

Create a new CachedInference backed by a freshly initialised cache.

pub fn run_or_cache<F>(&self, prompt: &str, run_inference: F) -> (String, bool)

where F: FnOnce() -> String,

Return a cached response if one exists, otherwise invoke run_inference and store its result.

Returns

(response, was_cache_hit) — the response string and whether it came from the cache.