Expand description
The descriptor limit: raised once (PID 1 to a million, anyone else to its
hard limit), and the open-repository handle cache sized to fit under it.
The fix for the rig’s “202nd repository” EMFILE (2026-08-29).
The descriptor limit: raised once, read once, and the handle cache sized
to fit under it.
§The bug this is written against — MEASURED 2026-08-29 on the rig
The 202nd repository touched since boot failed on every filesystem shape:
opening the git store at /data/repos/rig/z…-202.git/gunnar-git.znippy:
open journal …/objects.pack.journal: No file descriptors available (os error 24)and with it the ssh accept loop died — every later connection got
Connection reset by peer until reboot while gRPC kept answering. Two
facts made it: an open znippy store held five descriptors
(objects.pack ×3, objects.pack.journal, objects.tail — read off
/proc/self/fd in tests/fd_budget.rs; three since 2026-08-30, see
FDS_PER_OPEN_REPO) and the handle cache kept
DEFAULT_OPEN_REPOS = 256 of them,
while PID 1 runs with the kernel’s default RLIMIT_NOFILE soft limit of
1024 — nothing in the server or in korp-start ever raised it. 256 × 5
is 1280: the cache alone was over the limit, and at ~200 repositories the
next open, and the next accept, had nothing left.
§Both halves
nofileraises the soft limit once per process — as PID 1 (which holdsCAP_SYS_RESOURCE) toPID1_NOFILEon both soft and hard, elsewhere to whatever the hard limit already allows. Pure Rust:rustix::process. A limit that is still underCOMFORTABLE_NOFILEafterwards is aWARNongunnar::limits, which the dashboard shows as a row.open_repo_budgetsizes the handle cache from the limit that resulted:(soft − RESERVED_FDS) / FDS_PER_OPEN_REPO, clamped to[MIN_OPEN_REPOS, DEFAULT_OPEN_REPOS]. Under 1024 that is 256 handles (765 descriptors) with 256 left for listeners, sessions, the control database and the log — and the 1500-repository test passes underulimit -n 1024. Under a raised limit it is the old 256.
The listener half — an accept that fails with EMFILE must not end the
accept loop — is gunnar-ssh’s and lives in its server.rs.
Structs§
- Nofile
- The descriptor limit this process ended up with.
Constants§
- COMFORTABLE_
NOFILE - Below this many descriptors the server says so on the dashboard. 65 536 is
what a distribution’s
LimitNOFILE=for a network daemon typically reads; 1024 is whatsshdon a laptop gets and is where the bug lived. - FDS_
PER_ OPEN_ REPO - Descriptors one open repository handle costs on the znippy engine — read
off
/proc/self/fd(tests/fd_budget.rs, and per-store in the engine’s ownznippy-plugin-git/tests/fd_per_store.rs):objects.pack,objects.pack.journal,objects.tail. The gix arm holds fewer; budgeting for the larger of the two is the safe side. - MIN_
OPEN_ REPOS - The handle cache never shrinks under this, whatever the limit says: a
server that can keep no repository open re-opens (a
refold) on every request, and a limit that low is the WARN row’s job, not the cache’s. - PID1_
NOFILE - What PID 1 sets both limits to. The kernel’s
fs.nr_opendefault is 1 048 576 and asetrlimitabove it is refused, so this is the ceiling a stock kernel accepts; a smallernr_openis honoured by the fallback in [raise]. - RESERVED_
FDS - Descriptors kept back from the handle cache for everything that is not a cached repository: the two listeners, every live session (one socket, plus the transfer’s own store handle while it runs), the control database, the log tap, the scrape socket.
Functions§
- nofile
- The process’s descriptor limit, raised once.
- open_
repo_ budget - How many repository handles fit under a soft limit of
softdescriptors.