Module worker_pool 

A bounded set of persistent threads that a connection borrows rather than creates, closing the server-side per-connection thread leak.

The defect this closes

Every std::thread creation leaves 176–179 B behind permanently on RTEMS 6 — the thread’s TLS key is freed before its destructor runs, so the value block is never reclaimed. The cost is per creation, not per live thread, so a server that spawns a thread per accepted connection leaks without a ceiling: a client that connects and disconnects in a loop drains the target’s fixed heap for as long as the IOC runs.

DialPool closed the client dial path by this argument. This is the same argument on the serve side, and the two are separate primitives on purpose (doc/rtems-connection-worker-pool-design.md §3): a dial borrows one worker for a short job and queues over capacity; a connection borrows a set of workers for its whole life and is refused over capacity.

The unit of borrow is a set, not a worker

A PVA connection needs three threads together — the connection thread plus a reader pump plus a writer pump. If it could take two and block for the third, a server at capacity would deadlock: every connection holding two, each waiting on one nobody will free. So WorkerPool::acquire hands out a whole Worker array or nothing, and there is no API that borrows one worker on its own. The roster is heterogeneous within a set (the PVA set is one Big stack and two Small), which is exactly why a set is the unit and not N draws from N per-class pools — drawing separately would reopen the partial-borrow deadlock.

It does not raise the connection ceiling

A pooled connection occupies the same three stacks while it is live, because it is the same three threads doing the same work; the ceiling is per-connection memory (1,589,554 B measured per PVA connection on armv7-rtems-eabihf), not thread-creation residue. What the pool removes is the residue of the creation. Its other job is to be the single owner of connection admission — the one place that can refuse with EAGAIN.

The bound is memory, not just a count

A count bound cannot know when the target is out of thread memory: on x86_64-wrs-vxworks the CA pool’s count bound (141, derived from the descriptor budget) was never reached, because the process hit a reserved address-space ceiling at 46 concurrent clients first — and what happened there was not a refusal. pthread_create began to fail, then a std mutex lock returned EINVAL and killed a worker, then an allocation of 64 bytes failed and took the whole RTP down with signal 6. A bound that is reached after the target has run out is not an admission gate.

So every set’s memory is reserved from one process-wide budget before a thread is created (Reservation::try_reserve, POOL_RESERVATION_ENV), and a set that does not fit is refused with AcquireError::OutOfReservation while the target still has the memory to deliver the refusal. Process-wide because the resource is: an IOC runs several pools, and three pools each inside their own bound can still walk the process past the ceiling together. The count bounds stay exactly what they were — capacity is a descriptor bound for the CA server and an operator’s max_connections for the PVA server — because those are different resources and folding them into one number is what makes a bound unable to say which one ran out.

Accounting: busy is what is counted, and a set idles through one gate

A set returns to the idle pool when both its SetLease has dropped and every job dispatched on it has returned. running is incremented only by Worker::run/Worker::run_detached (the actor that really dispatched) and decremented only by the worker loop after the job’s closure has fully returned and been dropped (the actor that really finished). No side path pokes it. Every transition — lease drop, job completion — locks the set’s own state once, mutates, and checks the idle condition behind a parked flag so a double push is unrepresentable; only then, and never while holding the set lock, does it touch the pool lock to push the set back.

A worker that dies retires its set

That accounting is exact only while every dispatched job comes back, and a worker thread can die where the job’s catch_unwind does not reach — on target it did, at the memory wall, in a std mutex that returned EINVAL. So the set’s slot is released by the thread’s destructor (WorkerExit), not by a code path that a panic can skip: any exit that was not asked for marks the set dead, stops its siblings, and gives the slot back to created when the last of its threads is gone. A dead set is never pooled again, because a set one thread short cannot serve a connection — and a dispatch that lands on a worker already gone is reported as not run, never as a clean completion.

Structs

Job

A handle to a running job, joined on the borrower’s teardown path.

ObjectArenaExhausted

The target refused to materialise a kernel object a mutex needs.

SetLease

Proof that a set is borrowed. Its Drop is half of the return condition: the set cannot re-idle until this is gone and every job has finished.

ThreadCharge

One thread’s charge against the process account, held for exactly as long as the thread is.

Worker

One thread of a leased set, able to run exactly one job.

WorkerPool

A bounded, per-role set of persistent threads that connections borrow.

WorkerRole

One member of a worker set: how its thread is named, sized and banded.

Enums

AcquireError

Why WorkerPool::acquire refused.

Constants

POOL_RESERVATION_ENV

How many MiB of thread memory every pool in this process may reserve together. Overrides default_reservation_budget; read once, on first admission.