Struct ResponseCacheManager 

pub struct ResponseCacheManager { /* private fields */ }

Manages a filesystem cache of HTTP responses, keyed by request fingerprint.

The cache stores each response as a JSON file named after the hex-encoded SHA-1 fingerprint of the request that produced it. This is only active when the spider has development_mode enabled; production crawls skip the cache entirely.

Implementations

impl ResponseCacheManager

pub fn new(cache_dir: impl Into<PathBuf>) -> Self

Creates a new cache manager that stores entries in the given directory. The directory is created lazily on the first put call, so it does not need to exist at construction time.

pub fn get(&self, fingerprint: &[u8]) -> Option<Response>

Retrieves a cached response by its fingerprint, or None if not found.

The method reads the JSON file, deserializes the cached fields, and base64-decodes the response body. If any step fails (missing file, corrupt JSON, invalid base64), a warning is logged and None is returned so the engine falls through to a live fetch.

pub fn put( &self, fingerprint: &[u8], response: &Response, method: &str, ) -> Result<()>

Stores a response in the cache under the given fingerprint.

The response body is base64-encoded and the entire entry is serialized to JSON. The write is atomic: data goes to a temp file first, then is renamed to the final path, so a crash mid-write cannot corrupt an existing cache entry.

pub fn clear(&self) -> Result<()>

Removes all cached response files (.json) from the cache directory. Call this when you want to force a fresh crawl during development. Only files with a .json extension are deleted; other files in the directory are left untouched.