Expand description
Shared remote cache upload worker for spider and chromey.
This crate provides a high-performance, concurrent batch upload worker
for sending cached HTTP responses to a remote hybrid_cache_server.
§Client injection
Call set_client at startup, before the first upload or read, to
share your application’s reqwest::Client. This avoids compiling a
second TLS stack and reuses the existing connection pool. If nothing is
injected before first use, the crate builds a default client
(default_client_builder: 250 ms connect timeout, request timeout
equal to the batch timeout) and a later set_client is ignored.
client_source reports which one is active.
§Site keys
Every upload carries the job’s cache_site as the x-cache-site
header and as the payload’s website_key. A batch request only ever
holds jobs with the same cache_site.
§Features
- Batch-drain: Collects pending jobs from the channel before uploading
- Concurrent uploads: Configurable parallelism via semaphore (default 8)
- Lock-free dedup: DashSet for in-flight key tracking (no Mutex, no deadlocks)
- Rate limiting: Per-spawn pacing without blocking the drain loop
- Batch POST: Uses
/cache/index/batchwhen multiple jobs are ready - Best-effort:
try_enqueuenever blocks; drops on full queue - Metrics: with the default
metricsfeature, counters, a gauge and a histogram go through themetricsfacade (seemetrics)
Re-exports§
pub use client::build_payload;pub use client::client_source;pub use client::default_client_builder;pub use client::dump_batch_to_remote;pub use client::dump_site_batch_to_remote;pub use client::dump_to_remote;pub use client::get_client;pub use client::get_endpoint;pub use client::resolve_base_url;pub use client::set_client;pub use client::set_endpoint;pub use client::try_set_client;pub use client::ClientSource;pub use spool::spool_bytes;pub use spool::spool_dir;pub use spool::spool_max_bytes;pub use types::HttpVersion;pub use types::HybridCachePayload;pub use worker::default_max_body_size;pub use worker::default_max_concurrent;pub use worker::default_qps;pub use worker::default_queue_cap;pub use worker::default_queue_memory_budget;pub use worker::default_spool_writers;pub use worker::default_timeout_ms;pub use worker::enqueue;pub use worker::enqueue_best_effort;pub use worker::init_default_worker;pub use worker::init_remote_dump_worker;pub use worker::queue_bytes;pub use worker::set_skip_browser_dumps_enabled;pub use worker::set_spool_enabled;pub use worker::skip_browser_dumps_enabled;pub use worker::spool_enabled;pub use worker::try_enqueue;pub use worker::worker_inited;pub use worker::DumpJob;