Struct AppState 

pub struct AppState {
    pub config: Arc<AppConfig>,
    pub renderer: Arc<FallbackRenderer>,
    pub crawl_jobs: Arc<RwLock<HashMap<Uuid, CrawlJob>>>,
    pub extract_jobs: Arc<RwLock<HashMap<Uuid, ExtractRecord>>>,
    pub crawl_semaphore: Arc<Semaphore>,
    pub batch_pipeline_sem: Option<Arc<Semaphore>>,
    pub searxng: Option<Arc<SearxngClient>>,
    pub url_filter: Option<Arc<UrlFilterCfg>>,
}

Shared application state.

Fields

config: Arc<AppConfig>

renderer: Arc<FallbackRenderer>

crawl_jobs: Arc<RwLock<HashMap<Uuid, CrawlJob>>>

extract_jobs: Arc<RwLock<HashMap<Uuid, ExtractRecord>>>

/v2/extract jobs. Separate from crawl_jobs because an extract result is a single merged JSON object, not a Vec<ScrapeData>.

crawl_semaphore: Arc<Semaphore>

batch_pipeline_sem: Option<Arc<Semaphore>>

Process-wide cap on the total in-flight /v2/batch/scrape URL-pipelines across all batch-scrape jobs (aggregate bound so N jobs × width can’t explode). Targets batch scrape specifically because that’s the only wide fan-out: crawl is BFS-sequential and /v2/extract scrapes one URL at a time, both already bounded by the crawl_semaphore job cap. None = unbounded (config max_aggregate_batch_pipelines = 0/absent). Acquired as the first op in each batch URL future, before fetch.

searxng: Option<Arc<SearxngClient>>

SearXNG client. None when [search].searxng_url is unset, in which case /v1/search returns a clear search_disabled error.

url_filter: Option<Arc<UrlFilterCfg>>

Server-wide default /map URL filter. None disables the filter entirely (legacy behaviour). Per-request overrides may swap or extend this at handler time.

Implementations

impl AppState

pub fn new(config: AppConfig) -> CrwResult<Self>

pub async fn start_crawl_job(&self, req: CrawlRequest) -> Uuid

Start a new crawl job and return its UUID. Spawns a background task that acquires the crawl semaphore before running.

pub async fn start_batch_job( &self, urls: Vec<String>, template: ScrapeRequest, max_concurrency_override: Option<usize>, ) -> Uuid

Start a /v2/batch/scrape job over an explicit URL list and return its UUID. Reuses the crawl-job machinery (crawl_jobs + CrawlState) but scrapes the given URLs directly — no link discovery, no same-origin filtering, no dedup; input order is recoverable via metadata.sourceURL.

pub async fn start_extract_job( &self, entries: Vec<PreparedUrl>, template: ScrapeRequest, ) -> Uuid

Start an async extract job. Each entry is scraped with formats:[json] + the shared template; per-URL json objects are both (a) merged into one object for the FC-legacy data shape and (b) kept as an ordered per-URL array for the native /v1/extract contract. entries is in original request order and may include preflight-failed URLs (surfaced as failed results without being fetched).

pub async fn cancel_extract_job(&self, id: Uuid) -> CrwResult<ExtractRecord>

Request cancellation and return the persisted canonical state. Repeated calls are idempotent; terminal jobs are never rewritten.

pub async fn get_extract_job(&self, id: Uuid) -> CrwResult<ExtractRecord>

TTL-aware canonical lookup shared by v1, v2, and MCP handlers. A write lock makes expiry observation and removal one atomic operation.

Trait Implementations

impl Clone for AppState

fn clone(&self) -> AppState

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.