Function crawl_shared 

pub async fn crawl_shared(
    cfg: CrawlConfig,
    global: Arc<Semaphore>,
    on_progress: impl Fn(Progress) + Send + Sync,
) -> Result<CrawlOutput, CrawlError>

Crawls cfg.start_url, taking a slot of global for every request, so several crawls can share one in-flight cap. on_progress is called after each recorded page.