pub struct BrowserPool { /* private fields */ }Expand description
A pool of reusable browser tabs spread across one or more Chrome sessions.
Tabs are created lazily on first acquire(). Call
warmup() to optionally pre-create tabs for faster first
acquires.
§Usage
use void_crawl_core::pool::BrowserPool;
let pool = BrowserPool::from_env().await?;
// warmup() is optional — tabs are created on demand
let tab = pool.acquire().await?;
tab.page.navigate("https://example.com").await?;
let html = tab.page.content().await?;
pool.release(tab).await;
pool.close().await?;Implementations§
Source§impl BrowserPool
impl BrowserPool
Sourcepub fn new(config: PoolConfig, sessions: Vec<BrowserSession>) -> Self
pub fn new(config: PoolConfig, sessions: Vec<BrowserSession>) -> Self
Sourcepub async fn from_env() -> Result<Self>
pub async fn from_env() -> Result<Self>
Build a pool from environment variables.
| Variable | Description | Default |
|---|---|---|
CHROME_WS_URLS | Comma-separated ws:// or http:// URLs (connect mode) | — |
BROWSER_COUNT | Number of Chrome processes to launch | 1 |
TABS_PER_BROWSER | Max concurrent tabs per browser | 4 |
TAB_MAX_USES | Hard recycle threshold | 50 |
TAB_MAX_IDLE_SECS | Idle eviction timeout | 60 |
CHROME_NO_SANDBOX | Set to "1" to pass --no-sandbox | — |
CHROME_HEADLESS | Set to "0" for headful mode | 1 |
ACQUIRE_TIMEOUT_SECS | Max seconds to wait in acquire() | 30 |
VIEWPORT_WIDTH | Stealth viewport width | 1920 |
VIEWPORT_HEIGHT | Stealth viewport height | 1080 |
CDP_PORT_BASE | Pin Chrome’s --remote-debugging-port for launched browsers. Browser i uses base + i. Unset = OS-assigned (recommended; can’t conflict). | — |
Sourcepub async fn warmup(&self) -> Result<()>
pub async fn warmup(&self) -> Result<()>
Optionally pre-open tabs across all sessions and fill the ready queue.
Tabs are created in parallel across sessions, then inserted into the ready queue. Successful tabs are kept even when some fail; the first error (if any) is returned after all successful tabs are stored.
This is optional — if not called, tabs are created lazily on
first acquire().
Sourcepub async fn acquire(&self) -> Result<PooledTab>
pub async fn acquire(&self) -> Result<PooledTab>
Check out a tab from the pool.
If an idle tab is available, it is returned immediately. Otherwise,
a new tab is created on demand (up to tabs_per_browser * browsers
total). Blocks only when all tabs are currently in use.
Tabs that have exceeded tab_max_uses are silently hard-recycled.
The semaphore permit is returned on every error path so that failures never permanently shrink pool concurrency.
Sourcepub async fn acquire_timed(&self) -> Result<(PooledTab, u64)>
pub async fn acquire_timed(&self) -> Result<(PooledTab, u64)>
Like acquire, but also returns the milliseconds
spent blocked on the concurrency semaphore — the pure queueing wait,
excluding any lazy tab-creation latency that happens after a permit is
granted. A near-zero value means a slot was free immediately; a
non-trivial value means the caller queued behind other in-flight work.
Surfaced to MCP clients so an agent can tell when it has oversubscribed
the pool and should throttle or cap a batch at max_tabs.
Sourcepub async fn release(&self, tab: PooledTab)
pub async fn release(&self, tab: PooledTab)
Return a tab to the pool after use.
Instant return — no CDP round-trip on the common path. The next caller’s
navigate(url) overwrites prior page content; stealth scripts persist
across navigations.
The one exception: if a download was armed on this tab and never
completed (arm_download without a matching wait), we reset the CDP
download behavior before recycling so the next caller doesn’t inherit an
allowAndName pointing at a since-deleted quarantine dir. This costs
one CDP call only on the rare armed-but-abandoned path.
Sourcepub async fn evict_idle(&self) -> Result<()>
pub async fn evict_idle(&self) -> Result<()>
Close idle tabs that have exceeded tab_max_idle_secs and open fresh
replacements.
Intended to be called periodically from a background tokio task.
Sourcepub fn config(&self) -> &PoolConfig
pub fn config(&self) -> &PoolConfig
Access the pool configuration.
Sourcepub fn available_permits(&self) -> usize
pub fn available_permits(&self) -> usize
Free concurrency permits right now — how many more tabs could be
acquired without queueing. max_tabs - available_permits() is the
count currently checked out (in-flight fetches plus any held session
tabs). A live snapshot an agent can read via pool_status to size a
fan-out before submitting it.
Sourcepub fn start_eviction_task(self: Arc<Self>)
pub fn start_eviction_task(self: Arc<Self>)
Start a background Tokio task that periodically calls
evict_idle.
Idempotent — if an eviction task is already running, this is a
no-op. The handle is stored inside the pool and cancelled
automatically by close. Call
stop_eviction_task to cancel early.
§Panics
Must be called from within a Tokio runtime context.
Sourcepub fn stop_eviction_task(&self)
pub fn stop_eviction_task(&self)
Stop the background eviction task (if running).
Called automatically by close.
Trait Implementations§
Auto Trait Implementations§
impl !Freeze for BrowserPool
impl !RefUnwindSafe for BrowserPool
impl !UnwindSafe for BrowserPool
impl Send for BrowserPool
impl Sync for BrowserPool
impl Unpin for BrowserPool
impl UnsafeUnpin for BrowserPool
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> FmtForward for T
impl<T> FmtForward for T
Source§fn fmt_binary(self) -> FmtBinary<Self>where
Self: Binary,
fn fmt_binary(self) -> FmtBinary<Self>where
Self: Binary,
self to use its Binary implementation when Debug-formatted.Source§fn fmt_display(self) -> FmtDisplay<Self>where
Self: Display,
fn fmt_display(self) -> FmtDisplay<Self>where
Self: Display,
self to use its Display implementation when
Debug-formatted.Source§fn fmt_lower_exp(self) -> FmtLowerExp<Self>where
Self: LowerExp,
fn fmt_lower_exp(self) -> FmtLowerExp<Self>where
Self: LowerExp,
self to use its LowerExp implementation when
Debug-formatted.Source§fn fmt_lower_hex(self) -> FmtLowerHex<Self>where
Self: LowerHex,
fn fmt_lower_hex(self) -> FmtLowerHex<Self>where
Self: LowerHex,
self to use its LowerHex implementation when
Debug-formatted.Source§fn fmt_octal(self) -> FmtOctal<Self>where
Self: Octal,
fn fmt_octal(self) -> FmtOctal<Self>where
Self: Octal,
self to use its Octal implementation when Debug-formatted.Source§fn fmt_pointer(self) -> FmtPointer<Self>where
Self: Pointer,
fn fmt_pointer(self) -> FmtPointer<Self>where
Self: Pointer,
self to use its Pointer implementation when
Debug-formatted.Source§fn fmt_upper_exp(self) -> FmtUpperExp<Self>where
Self: UpperExp,
fn fmt_upper_exp(self) -> FmtUpperExp<Self>where
Self: UpperExp,
self to use its UpperExp implementation when
Debug-formatted.Source§fn fmt_upper_hex(self) -> FmtUpperHex<Self>where
Self: UpperHex,
fn fmt_upper_hex(self) -> FmtUpperHex<Self>where
Self: UpperHex,
self to use its UpperHex implementation when
Debug-formatted.Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
Source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> Pipe for Twhere
T: ?Sized,
impl<T> Pipe for Twhere
T: ?Sized,
Source§fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> Rwhere
Self: Sized,
fn pipe<R>(self, func: impl FnOnce(Self) -> R) -> Rwhere
Self: Sized,
Source§fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> Rwhere
R: 'a,
fn pipe_ref<'a, R>(&'a self, func: impl FnOnce(&'a Self) -> R) -> Rwhere
R: 'a,
self and passes that borrow into the pipe function. Read moreSource§fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> Rwhere
R: 'a,
fn pipe_ref_mut<'a, R>(&'a mut self, func: impl FnOnce(&'a mut Self) -> R) -> Rwhere
R: 'a,
self and passes that borrow into the pipe function. Read moreSource§fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R
fn pipe_borrow<'a, B, R>(&'a self, func: impl FnOnce(&'a B) -> R) -> R
Source§fn pipe_borrow_mut<'a, B, R>(
&'a mut self,
func: impl FnOnce(&'a mut B) -> R,
) -> R
fn pipe_borrow_mut<'a, B, R>( &'a mut self, func: impl FnOnce(&'a mut B) -> R, ) -> R
Source§fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R
fn pipe_as_ref<'a, U, R>(&'a self, func: impl FnOnce(&'a U) -> R) -> R
self, then passes self.as_ref() into the pipe function.Source§fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R
fn pipe_as_mut<'a, U, R>(&'a mut self, func: impl FnOnce(&'a mut U) -> R) -> R
self, then passes self.as_mut() into the pipe
function.Source§fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R
fn pipe_deref<'a, T, R>(&'a self, func: impl FnOnce(&'a T) -> R) -> R
self, then passes self.deref() into the pipe function.Source§impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> Read<Exclusive, BecauseExclusive> for Twhere
T: ?Sized,
Source§impl<T> Tap for T
impl<T> Tap for T
Source§fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self
fn tap_borrow<B>(self, func: impl FnOnce(&B)) -> Self
Borrow<B> of a value. Read moreSource§fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self
fn tap_borrow_mut<B>(self, func: impl FnOnce(&mut B)) -> Self
BorrowMut<B> of a value. Read moreSource§fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self
fn tap_ref<R>(self, func: impl FnOnce(&R)) -> Self
AsRef<R> view of a value. Read moreSource§fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self
fn tap_ref_mut<R>(self, func: impl FnOnce(&mut R)) -> Self
AsMut<R> view of a value. Read moreSource§fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self
fn tap_deref<T>(self, func: impl FnOnce(&T)) -> Self
Deref::Target of a value. Read moreSource§fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self
fn tap_deref_mut<T>(self, func: impl FnOnce(&mut T)) -> Self
Deref::Target of a value. Read moreSource§fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self
fn tap_dbg(self, func: impl FnOnce(&Self)) -> Self
.tap() only in debug builds, and is erased in release builds.Source§fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self
fn tap_mut_dbg(self, func: impl FnOnce(&mut Self)) -> Self
.tap_mut() only in debug builds, and is erased in release
builds.Source§fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self
fn tap_borrow_dbg<B>(self, func: impl FnOnce(&B)) -> Self
.tap_borrow() only in debug builds, and is erased in release
builds.Source§fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self
fn tap_borrow_mut_dbg<B>(self, func: impl FnOnce(&mut B)) -> Self
.tap_borrow_mut() only in debug builds, and is erased in release
builds.Source§fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self
fn tap_ref_dbg<R>(self, func: impl FnOnce(&R)) -> Self
.tap_ref() only in debug builds, and is erased in release
builds.Source§fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self
fn tap_ref_mut_dbg<R>(self, func: impl FnOnce(&mut R)) -> Self
.tap_ref_mut() only in debug builds, and is erased in release
builds.Source§fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self
fn tap_deref_dbg<T>(self, func: impl FnOnce(&T)) -> Self
.tap_deref() only in debug builds, and is erased in release
builds.