Shared remote cache upload worker for spider and chromey.

This crate provides a high-performance, concurrent batch upload worker for sending cached HTTP responses to a remote hybrid_cache_server.

Features

• Batch-drain: Collects pending jobs from the channel before uploading
• Concurrent uploads: Configurable parallelism via semaphore (default 32)
• Lock-free dedup: DashSet for in-flight key tracking (no Mutex, no deadlocks)
• Rate limiting: Per-spawn pacing without blocking the drain loop
• Batch POST: Uses /cache/index/batch when multiple jobs are ready
• Best-effort: try_enqueue never blocks; drops on full queue

Quick start

use spider_remote_cache::{DumpJob, HttpVersion};

# async fn example() {
let job = DumpJob {
    cache_key: "my-key".into(),
    cache_site: "example.com".into(),
    url: "https://example.com/page".into(),
    method: "GET".into(),
    status: 200,
    request_headers: Default::default(),
    response_headers: Default::default(),
    body: b"<html>hello</html>".to_vec(),
    http_version: HttpVersion::Http11,
    dump_remote: None,
};

// Best-effort enqueue (never blocks, drops if full).
spider_remote_cache::enqueue_best_effort(job).await;
# }