Struct WorkerClientPool 

pub struct WorkerClientPool { /* private fields */ }

Connection pool for WorkerClient instances.

Caches authenticated gRPC channels by worker address, avoiding the overhead of re-establishing connections and re-authenticating for every block I/O. Matches Java’s FileSystemContext.acquireBlockWorkerClient() pattern.

The pool is thread-safe and can be shared across concurrent workers.

Single-Flight Reconnect

When a SASL stream silently expires server-side, many concurrent RPCs on the same cached channel will fail simultaneously with UNAUTHENTICATED. Without coordination each observer would independently invoke reconnect, producing a “thundering herd” that serialises through the pool’s write lock and wastes CPU/RTT on duplicate TCP+SASL handshakes.

To collapse this herd, each WorkerClient carries a monotonic generation tag. Callers pass the observed generation back into reconnect_if_stale after an auth failure; only the first observer of a given generation actually performs the reconnect, all other concurrent observers receive the already-replaced client. This reduces N concurrent reconnects to exactly 1.

Implementations

impl WorkerClientPool

pub fn new(config: GoosefsConfig) -> Self

Create a new empty connection pool.

pub fn pool_size(&self) -> usize

Number of channels pooled per worker address.

pub async fn acquire(&self, addr: &str) -> Result<WorkerClient>

Acquire a WorkerClient for the given address.

Returns a cached client if one exists, otherwise creates a new connection. With pool_size > 1 this round-robins across the per-worker channels so concurrent block reads spread over multiple HTTP/2 connections. The tonic Channel itself also multiplexes, so each channel handles many concurrent RPCs.

pub async fn invalidate(&self, addr: &str)

Remove a worker from the pool (e.g., after a connection failure).

The next acquire() call for this address will create a fresh connection.

Also drops the per-address reconnect mutex from reconnect_locks so the map does not grow unbounded when many distinct worker addresses come and go (e.g. worker scale-up / scale-down). It is safe to drop the mutex here because:

• any caller that already holds an Arc<AsyncMutex<()>> clone keeps it alive for the duration of its critical section;
• the next reconnect_lock_for() call for the same address will lazily install a fresh mutex. In the worst case two callers racing across an invalidate() may hold different mutex instances briefly, but the cache double-check inside reconnect_if_stale still serialises the actual handshake via the generation comparison, so correctness is preserved.

pub async fn reconnect_if_stale( &self, addr: &str, stale_generation: u64, ) -> Result<WorkerClient>

Single-flight reconnect: invalidate + reconnect only if the currently cached client’s generation still matches stale_generation.

This is the preferred recovery path on authentication failure. The caller passes the generation() of the client that just failed; because every WorkerClient carries a unique monotonic generation allocated by this pool:

• If another concurrent task has already reconnected in response to the same underlying SASL expiry, the cached generation will have advanced past stale_generation and this call returns the already-replaced client without performing another TCP+SASL handshake.
• Otherwise, this call performs exactly one reconnect under the per-address mutex.

Net effect: N concurrent AuthenticationFailed observers on the same channel trigger exactly one reconnect instead of N.

pub async fn reconnect(&self, addr: &str) -> Result<WorkerClient>

Invalidate a cached worker connection and immediately reconnect.

Prefer reconnect_if_stale whenever the caller holds a reference to the failing WorkerClient — it deduplicates concurrent reconnects triggered by the same underlying SASL expiry.

This unconditional variant is kept for paths where the caller does not know the generation of the failing client (e.g. a stand-alone connect() failure that never produced a WorkerClient). It acquires the same per-address reconnect mutex so it still coalesces against any in-flight reconnect_if_stale.

pub fn new_shared(config: GoosefsConfig) -> Arc<Self>

Create a new pool wrapped in Arc for shared ownership.