# net-io - Project Plan
> The design, strategy, and decision record for `net-io`. This is a living document: any design decision, idea, or open question about net-io that comes up anywhere (chat, review, a sibling crate) is written here or in [`ROADMAP.md`](./ROADMAP.md) before it is acted on, so nothing lives only in someone's memory.
>
> **Read order:** [`../REPS.md`](../REPS.md) (portfolio standard) → [`DIRECTIVES.md`](./DIRECTIVES.md) (engineering law for this crate) → **this file** → [`ROADMAP.md`](./ROADMAP.md) (phases and exit criteria) → [`../docs/API.md`](../docs/API.md) (the locked public API) → [`SYS-SURFACE.md`](./SYS-SURFACE.md) (every OS call, constant, and struct the crate declares).
>
> **Status:** `0.2.0` design lock. The public API is written in `docs/API.md`; this plan records the design behind it and every decision (section 14).
---
## Contents
1. [Purpose](#1-purpose)
2. [Where net-io sits](#2-where-net-io-sits)
3. [Scope](#3-scope)
4. [Design principles](#4-design-principles)
5. [Requirements](#5-requirements)
6. [Architecture](#6-architecture)
7. [Deployment modes](#7-deployment-modes)
8. [Security](#8-security)
9. [Performance](#9-performance)
10. [Ecosystem and dependencies](#10-ecosystem-and-dependencies)
11. [Testing and verification](#11-testing-and-verification)
12. [Open decisions](#12-open-decisions)
13. [Ideas parking lot](#13-ideas-parking-lot)
14. [Decision log](#14-decision-log)
15. [Spikes at the design lock](#15-spikes-at-the-design-lock)
Appendix A: [edge cases](#appendix-a-edge-cases)
---
## 1. Purpose
net-io is the entrance and the exit. Every byte that reaches one of our services arrives through it, and every byte a service sends leaves through it. It owns:
1. **Sockets and their options:** TCP, UDP, Unix domain sockets (Unix, and Windows through AF_UNIX), and Windows named pipes; every socket option a server needs, set through its own small FFI layer because `std` exposes few of them.
2. **The life of each connection:** accepted, admitted, handshaking, open, parked, draining, closed, with exactly one `CloseReason` per connection and a `ConnectionId` that is never reused inside a process.
3. **The limits that stop a peer from holding what it should not:** global, per-listener, and per-source connection caps; handshake, idle, read, write, and total deadlines; a minimum transfer rate against slowloris and slow-read; bounded buffers; bounded queues.
4. **The drivers that move the bytes:** a bounded blocking worker pool first (works everywhere, including shared hosting and CI), its own reactor second (epoll on Linux, kqueue on macOS and FreeBSD, IOCP on Windows), and io_uring later on Linux, all behind one completion-shaped driver interface.
5. **The seams other crates plug into without net-io depending on them:** a gate (admission at accept, a check before every write), a sans-IO TLS seam, an event sink, a control line, and a completion queue for work that left the connection's worker.
6. **Graceful drain and zero-downtime restarts:** a door that opens last and closes first, drain with a deadline, systemd socket activation, FastCGI descriptor-0 inheritance, and listener hand-off between processes by descriptor passing.
7. **The real client address behind a proxy:** PROXY protocol v1 and v2 parsed sans-IO, honoured only from configured trusted proxies.
It is written once and used by everything that listens on a port or dials one: the web stack, a database wire protocol, raft-io's transport, rpc-io over TCP, admin consoles, back-channel clients.
What it is not:
| Not | Why | Where it lives |
|---|---|---|
| an HTTP server or any protocol parser | protocol-free rule (2.2) | `http-net`, `ws-net`, `fcgi-net`, `wire-codec`, a database wire protocol |
| a TLS implementation | TLS is never written in-house (D-003, D-012) | `tls-net` (adapter over rustls) |
| a policy engine (bans, scores, rates per prefix) | policy belongs to the gate | `bouncer-io`, `sentinel-sec` |
| an async runtime or task scheduler | no rented runtime (D-004) | an executor crate, through a driver seam |
| a DNS resolver | `std::net::ToSocketAddrs` until a real resolver exists (D-002) | `dns-net` (later) |
| QUIC | its own transport with TLS 1.3 built in (D-002) | `quic-net` (later), on net-io's UDP socket |
| a signal handler | signals are translated by the signal library and decided by the application | `signal-mod`; net-io exposes `close_door`, `drain`, `shutdown`, `stop`, and the control line |
| a supervisor or watchdog | a separate process | the application's supervisor |
| a privilege dropper | a verified security control | `shield-sec`, called after net-io has bound the sockets |
---
## 2. Where net-io sits
### 2.1 The request path
Each stage rejects as early and as cheaply as it can. An IP ban costs nanoseconds; a password check costs milliseconds and a database lookup. A flood that the door can stop must never reach the expensive stages.
```
inbound outbound
┌───────────────────────┐ ┌───────────────────────┐
│ net-io: accept │ socket, ConnectionId, │ net-io: write │
│ │ peer address, caps │ │
├───────────────────────┤ ├───────────────────────┤
│ bouncer-io: door │ bans, budgets │ bouncer-io: egress │ Gate::check before
│ (Gate::admit) │ (before any byte is read) │ (Gate::check) │ every write
├───────────────────────┤ ├───────────────────────┤
│ PROXY header, then │ real client, readmission │ customs-sec: export │ only declared data
│ Gate::readmit │ │ │ leaves
├───────────────────────┤ ├───────────────────────┤
│ TLS (tls-net adapter) │ handshake, fingerprint │ handlers / services │ bouncer checkpoints
├───────────────────────┤ │ │ between stages
│ protocol engine │ http-net / ws-net / └───────────────────────┘
│ (Session) │ fcgi-net / a DB wire ▲
├───────────────────────┤ │
│ customs-sec: head │ declared size, type, route │
├───────────────────────┤ │
│ gate-net: authn │ who are you (session/login) │
├───────────────────────┤ │
│ gate-sec: authz │ may you do this │
├───────────────────────┤ │
│ customs-sec: body │ payload matches the contract ──────────┘
└───────────────────────┘
underneath everything: shield-sec (hardening) watching everything: sentinel-sec (detection)
```
### 2.2 The protocol-free rule
A crate is **protocol-free** when it does its job without knowing the format of the messages crossing the wire. net-io moves bytes and knows who is connected; it never parses HTTP. The same is true of bouncer-io, gate-net, and session storage. Only thin adapters at the top know where a protocol keeps its credentials or how it phrases a refusal (in net-io's terms: the `Session` writes the denial text for a code the gate gives).
**Litmus test:** delete `http-net` from the workspace. Every protocol-free crate still compiles; only the web crates break.
### 2.3 Who uses it
| Consumer | What it needs from net-io |
|---|---|
| `serve-net` (network service module) | listeners per ingress mode, sessions over every driver, limits, drain, the gate seam, PROXY, the TLS seam |
| a private request pipeline built on `serve-net` | everything above, plus `Input::swap` for request bodies and WebSocket messages, the per-worker `Completer` for offloaded work, lanes at admission, the control line, door order, hand-off with a supervisor process, dialed back-channel connections registered like accepted ones |
| `bouncer-io` | implements `Gate`; uses `ServerHandle::kill`, `ConnectionInfo`, `ConnectionId`, `Cidr`, `PrefixKey`, `normalize` |
| `tls-net` | implements `TlsAcceptor` and `TlsSession` |
| `fetch-net`, `proxy-net`, `smtp-net`, back-channel clients | outbound connect with timeouts, socket options, the same limits, `set_gate` |
| A database's network layer, `raft-io`, `rpc-io` | TCP listener and client, drain, limits, hand-off |
| `quic-net` (later) | UDP socket, batch IO, GSO/GRO, packet info |
| an executor crate | net-io implements its driver trait so workers park inside the reactor (feature `executor`, D-030) |
### 2.4 Dependency direction
```
bouncer-io ──▶ net-io ◀── tls-net (implementers depend on net-io; never the reverse: D-006)
serve-net ──▶ net-io, bouncer-io, http-net
net-io ──▶ (nothing at run time) executor adapter: optional feature, D-030
```
---
## 3. Scope
### 3.1 In scope
| Area | Contents | Grade summary |
|---|---|---|
| TCP | listener, accepted connection, outbound connect (timeout, local bind, options), half-close, abortive close, vectored IO, peek | P0 |
| Socket options | `TCP_NODELAY`; keepalive with idle, interval, count; `SO_LINGER`; buffer sizes; `SO_REUSEADDR` / `SO_EXCLUSIVEADDRUSE`; `SO_REUSEPORT`; `IPV6_V6ONLY`; backlog; `TCP_DEFER_ACCEPT`; TCP Fast Open; `SO_INCOMING_CPU`; `TCP_USER_TIMEOUT` / `TCP_MAXRT`; `TCP_QUICKACK`; `TCP_NOTSENT_LOWAT`; TTL and DSCP | P0 to P3 per option (`docs/API.md`, option table) |
| UDP | bound socket, send and receive with source address, truncation detection, per-source budgets, no-amplification guard, batch IO, packet info, GSO/GRO | P1 (batch and offloads P2/P3) |
| Local sockets | Unix domain sockets on Unix and Windows (AF_UNIX, 1803+); Windows named pipes; permissions; stale socket handling; peer credentials | P1 (Windows AF_UNIX P2) |
| Addresses | normalization (IPv4-mapped IPv6 to IPv4), `Cidr`, `PrefixKey` (IPv4 /32 and /24, IPv6 /128, /64, /56, /48), `TrustedProxies`, endpoints | P0 |
| Identity | `ConnectionId`, `ListenerId`, `WorkerId`, `Lane`, `ConnectionInfo` | P0 |
| Lifecycle | states, `CloseReason`, exactly one closed event | P0 |
| Limits | caps; backlog; handshake, idle, read, write, total, park deadlines; minimum transfer rate; buffer and queue bounds; adaptive deadlines under pressure | P0 (adaptive P2) |
| Accept discipline | batched accept, reserve descriptor on `EMFILE`/`ENFILE`, accept pause with backoff, transient error classes | P0 |
| Drivers | blocking pool (0.3), reactor: epoll, kqueue, IOCP (0.5), io_uring (after 1.0, P3); one completion-shaped interface; per-worker completion queue | P0 (io_uring P3) |
| Buffers | per-worker pools in size classes, bounded total, zero-copy read views, O(1) buffer swap, wipe on release, vectored writes, `sendfile` / `TransmitFile` | P0 to P2 |
| Timers | hierarchical timing wheel per worker, lazy deadline extension, tickless idle | P0 |
| Server | door, drain with deadline, shutdown, wait/stop, adopt dialed connections, kill by id, stats, worker placement | P0 (placement P2, NUMA P3) |
| Hooks | `Gate`, `KillHandle`, `EventSink`, `ControlLine`, `Completer`, TLS seam | P0 |
| PROXY protocol | v1 and v2 parsing (TLVs, CRC32C, SSL sub-TLVs), off or required, trusted peers only, encoder for outbound | P0 parsing, P2 encoder |
| Restarts | systemd `LISTEN_FDS`, FastCGI descriptor 0, listener hand-off by descriptor passing, raw adoption | P0 hand-off, P1 inheritance |
| Outbound | connect with timeout, connect to a host name through `std` resolution, Happy Eyeballs (RFC 8305, simplified) | P0 connect, P2 Happy Eyeballs |
| Platform | Linux, macOS, Windows first class; FreeBSD for the reactor; close-on-exec everywhere; no `SIGPIPE` | P0 |
### 3.2 Out of scope, and where it goes
| Not in net-io | Goes to | Reason |
|---|---|---|
| QUIC, HTTP/3 | `quic-net` (later) | a large separate job; uses net-io's UDP, GSO/GRO, packet info |
| DNS resolution beyond `std` | `dns-net` (later) | net-io accepts resolved addresses; `connect_host` uses `std` |
| the TLS implementation | `tls-net` | net-io defines the seam; kTLS is a parking-lot idea |
| ban lists, per-prefix rates, strikes, revocation | `bouncer-io` | net-io provides the gate seam, kill by id, and only a per-source concurrency cap (D-025) |
| HTTP keep-alive policy, `Connection: close`, 503 bodies | protocol layer | net-io exposes `pressure`, `draining`, and the denial and refusal slots; it never formats them |
| request pipelines, what a lane means | the application | net-io carries an opaque `Lane(u8)` and the gate's ticket |
| offload pools | an executor or the application | net-io only provides the per-worker `Completer` |
| process lock, umask, open-file limit raise | `sys-lib` | net-io reads the limit and refuses a configuration that exceeds it |
| privilege drop | `shield-sec` | called after net-io binds and before `open_door` |
| signal handling | `signal-mod`, the application | |
| eBPF/XDP ban export | `bouncer-io` parking lot | net-io may later expose `SO_ATTACH_REUSEPORT_*` and a raw descriptor accessor (Tier 3) |
| connection pooling for clients | `pool-mod`, `fetch-net` | net-io supplies connections |
| compression, framing | `wire-codec`, protocol crates | |
| payload and contract validation | `customs-sec` | needs the protocol and the route |
| authentication, sessions, authorization | `gate-net`, `gate-sec` | identity is not transport |
---
## 4. Design principles
1. **Ours, not rented.** Zero runtime dependencies. OS calls `std` lacks go through our own minimal FFI declarations in `src/sys/` (D-014), not `libc`, `windows-sys`, `mio`, `socket2`, or `tokio`.
2. **Unsafe is fenced.** `unsafe` exists only inside `src/sys/`, every block carries a `// SAFETY:` comment, and everything outside `sys/` is `#![deny(unsafe_code)]`. Until `sys/` lands (0.3), the crate stays `#![forbid(unsafe_code)]`.
3. **Hooks, not dependencies.** net-io defines traits (gate, TLS, event sink, control line, clock); implementers depend on net-io, never the reverse.
4. **Everything is bounded.** Every buffer, queue, count, and wait has a limit, checked before the resource is committed.
5. **Fail closed.** When a hook errors or panics, or a limit cannot be evaluated, the connection is refused or closed, never waved through.
6. **Cross-platform for real.** Linux, macOS, and Windows first class on stable and MSRV in CI; FreeBSD from the reactor. Platform differences documented per feature, never hidden.
7. **Measured, not guessed.** No performance claim without a benchmark at realistic scale; system calls are a hot-path cost like allocations.
8. **Simple first.** Tier 1 (a free function, one line) covers the common uses; builders are Tier 2; the generic, allocation-free surface is Tier 3 (`docs/API.md`, conventions).
---
## 5. Requirements
Each requirement has an id that tests and roadmap items cite. "Must" means the version that introduces the item does not ship without it.
### 5.1 Functional
**Addresses and identity**
- **FR-1** Normalize every peer address at accept: an IPv4-mapped IPv6 address becomes the IPv4 address; scope ids are kept for link-local IPv6; nothing else is rewritten (NAT64 is not translated).
- **FR-2** Parse and match CIDR blocks for IPv4 and IPv6, rejecting host bits beyond the prefix unless the caller asks for masking.
- **FR-3** Produce prefix keys: IPv4 /32 and /24; IPv6 /128, /64, /56, /48; each a fixed-size `Copy` value usable as a hash key.
- **FR-4** `TrustedProxies`: an immutable set of at most 1,024 CIDR blocks with an allocation-free lookup.
- **FR-5** Issue a `ConnectionId` for every accepted, adopted, dialed-and-registered, or inherited connection; never reused within a process, never zero.
- **FR-6** Build a `ConnectionInfo` for every connection: local and peer endpoints, real peer (after PROXY), transport, origin, listener, worker, lane, accept instant, peer credentials (local sockets), TLS facts (after the handshake).
**TCP**
- **FR-7** Bind a TCP listener with options applied before `bind` (reuse address, exclusive address use on Windows, reuse port, IPv6-only) and before `listen` (backlog, defer accept, fast open queue).
- **FR-8** Accept in batches up to a configured count per wake-up; set non-blocking (reactor), close-on-exec, no-SIGPIPE, and the configured per-connection options on each accepted socket.
- **FR-9** Survive descriptor exhaustion at accept: on `EMFILE`/`ENFILE` (Windows `WSAEMFILE`, `WSAENOBUFS`), release the reserve descriptor, accept and reset one pending connection, reopen the reserve, emit `DescriptorPressure`, and pause accepting with exponential backoff (1 ms doubling to 1 s) instead of spinning.
- **FR-10** Treat `ECONNABORTED`, `EPROTO`, `EPERM`, `ENOPROTOOPT`, `EHOSTDOWN`, `ENONET`, `EHOSTUNREACH`, `EOPNOTSUPP`, `ENETUNREACH` from `accept` as per-connection failures (counted, accept continues); `ENOBUFS`/`ENOMEM` as pressure (pause); `EBADF`, `EINVAL`, `ENOTSOCK` as listener failure (listener closed, event, error).
- **FR-11** Connect outbound with a deadline, optional local bind, and options; report `ConnectTimeout`, `ConnectionRefused`, `HostUnreachable`, `NetworkUnreachable` distinctly.
- **FR-12** Read into caller or pooled buffers; write single and vectored buffers; continue partial writes from the exact offset until done, the deadline passes, or the gate denies.
- **FR-13** Half-close: `shutdown(Write)` sends FIN and keeps reading; `shutdown(Read)` stops reading.
- **FR-14** Abortive close: `SO_LINGER {on, 0}` and close (RST, resources freed at once); used by kill, gate denial, refusal, and the drain deadline.
- **FR-15** Graceful close: flush within the write deadline, FIN, read and discard inbound bytes for a bounded linger (2 s, 64 KiB), then close.
**Socket options**
- **FR-16** Set and get every option of the option table on every platform that supports it; elsewhere return `Unsupported { feature, platform }`, never silently ignore, unless the option is a documented hint.
- **FR-17** Report the effective value where the OS adjusts it (buffer sizes doubled on Linux; backlog capped by `somaxconn`).
**Limits and timeouts**
- **FR-18** Enforce global, per-listener, and per-source concurrency caps (keyed by `PrefixKey`, /32 and /64 by default) before any byte is read; the per-source table has capacity above the global cap, so it can never overflow.
- **FR-19** One handshake deadline covers the PROXY header and the TLS handshake together.
- **FR-20** Idle deadline while idle; read and write deadlines while bytes are expected or pending.
- **FR-21** A total deadline per exchange set by the session.
- **FR-22** A minimum transfer rate per direction while it is active: after a grace period, the bytes moved over a sliding window must reach the configured rate, or the connection closes with `SlowPeer { direction }`.
- **FR-23** Bound every buffer and queue: read buffer class per connection, output bytes per connection, completion queue per worker, accept batch, buffer budget.
- **FR-24** Under pressure, shrink handshake and idle deadlines by a configured factor (P2).
- **FR-25** Validate every limit at build time; reject zero, out-of-range, and inconsistent combinations with `InvalidConfig { field, reason }`.
**Gate**
- **FR-26** Call `Gate::admit` after accept and before any read; call `Gate::readmit` after a trusted PROXY header reveals the real peer; on refusal close abortively without reading, optionally after one non-blocking write of at most 512 refusal bytes.
- **FR-27** Store the lane and the ticket admission returns; pass the ticket to every later gate call and back to `Gate::closed` exactly once.
- **FR-28** Call `Gate::check` before every write system call (each chunk of a vectored or file send, each TLS flush); on denial overwrite all pending output with zeroes, write the session's denial if no byte of the current response has left, then close abortively.
- **FR-29** A `KillHandle` per connection, `Send + Sync + Clone`, closes the connection abortively from any thread, wakes any blocked or pending operation, and yields exactly one closed event; killing a closed connection returns `false`. `ServerHandle::kill` does the same by id.
- **FR-30** Typed, fixed-size events to an `EventSink` for listener, door, accept, refusal, admission, PROXY resolution, TLS completion, limit hits, kills, gate denials, close (reason, bytes, duration), accept errors, descriptor pressure, drain, control-line reactions, driver fallback.
- **FR-31** Read the `ControlLine` at every worker loop iteration and before every accept batch and admission; react to `Shed`, `Lockdown`, `Draining`, `Stopping` (6.12); wake every worker on demand.
- **FR-32** A `Completer` per worker: a bounded queue other threads use to post `(ConnectionId, token, value)` to the owning worker and wake it; a full queue returns `PostError::Full` and never blocks.
**Sessions and buffers**
- **FR-33** Drive a sans-IO `Session` per connection with the same code on every driver.
- **FR-34** Give `on_data` a zero-copy view of the bytes read; let the session consume a prefix, keep the rest, or swap the whole buffer with a caller-owned buffer of the same class in O(1).
- **FR-35** Let a session park its connection (no reads, no idle timer, a park deadline) until a completion arrives or the park deadline passes.
- **FR-36** Wipe the used part of every buffer when it returns to a pool, unless configured off.
- **FR-37** Send files with `sendfile` / `TransmitFile`, falling back to read-and-write where the platform or TLS prevents it, with a gate check before each chunk.
**Server, drain, restarts**
- **FR-38** Separate binding from accepting: `build` binds or adopts and starts workers with the door closed; `open_door` starts accepting; `close_door` stops accepting within one loop iteration while keeping the listening sockets open.
- **FR-39** Drain: after `close_door`, tell every session to finish, close idle connections at once, wait for the rest until the deadline, then kill the remainder with `DrainDeadline`; return a `DrainReport`.
- **FR-40** Shutdown: close the door, drain, close the listening sockets (unless released), join every worker; afterwards no net-io thread runs and every descriptor net-io opened is closed. `wait` blocks until a stop is requested, then shuts down.
- **FR-41** Release listeners for hand-off and send them to another process (`SCM_RIGHTS` on Unix; `WSADuplicateSocketW` blobs over an authenticated named pipe on Windows) with a versioned, checksummed manifest; the receiver verifies every socket and refuses all of them if any check fails.
- **FR-42** Inherit listeners from systemd (`LISTEN_PID`, `LISTEN_FDS`, `LISTEN_FDNAMES`, `LISTEN_PIDFDID`, descriptors from 3), validating the pid and each descriptor.
- **FR-43** Detect and adopt the FastCGI listening socket on descriptor 0 (Unix).
- **FR-44** Adopt an outbound connection into a running server so it is registered exactly like an accepted one (`AdmitStage::Adopt`).
**PROXY protocol**
- **FR-45** Parse PROXY v1 (at most 107 bytes) and v2 sans-IO from any split of the input, reporting how many bytes were consumed.
- **FR-46** Accept a header only on listeners configured `Required` and only from peers in `TrustedProxies`; never guess whether a header is present; an untrusted peer, a missing header, or an invalid header closes the connection with a distinct reason.
- **FR-47** Handle v2 `LOCAL`, `UNSPEC`, TCP and UDP over IPv4 and IPv6, and UNIX; verify CRC32C when present; expose ALPN, authority, unique id (at most 128 bytes), SSL sub-TLVs, and NETNS as borrowed views; skip unknown and `NOOP` TLVs within bounds.
- **FR-48** Encode v1 and v2 headers for outbound use (P2).
**UDP**
- **FR-49** Bind, send to, receive from, connect; report truncated datagrams instead of returning them silently cut.
- **FR-50** Per-source and global datagram and byte budgets with a bounded source table.
- **FR-51** No amplification: to a source not marked verified, bytes sent never exceed a configured factor (default 1) of bytes received from it within the window.
- **FR-52** Batch receive and send, packet info (P2); GSO/GRO (P3).
**Local sockets**
- **FR-53** Bind Unix sockets with a configured mode (default `0o600`, `0o660` when a group is named) and optional owner and group; refuse a path longer than the platform limit; remove a stale socket file only if `lstat` says socket and a connect to it is refused.
- **FR-54** Linux abstract names only with a mandatory peer uid check (P3).
- **FR-55** Report peer credentials on every platform that has them.
- **FR-56** Pass descriptors over Unix sockets with close-on-exec, detect control-message truncation, and close every descriptor of a failed exchange.
- **FR-57** Windows named pipes with an owner-only security descriptor by default, `PIPE_REJECT_REMOTE_CLIENTS`, and `FILE_FLAG_FIRST_PIPE_INSTANCE`; clients connect with `SECURITY_SQOS_PRESENT | SECURITY_IDENTIFICATION`.
**TLS seam**
- **FR-58** Drive a sans-IO TLS session in place: feed ciphertext in the connection's read buffer, take plaintext from the same buffer, encrypt into the output buffer; finish the handshake under the handshake deadline; send `close_notify` on graceful close.
- **FR-59** The engine inspects the ClientHello inside `step` and may refuse it; net-io closes such a connection with `HelloRefused` (D-019).
- **FR-60** Surface SNI, ALPN, version, and cipher suite in `ConnectionInfo` after the handshake.
**Platform and process**
- **FR-61** At build, read the open-descriptor limit and refuse a configuration whose caps plus reserves exceed it.
- **FR-62** Create every descriptor and handle close-on-exec / non-inheritable; where the OS has no atomic form (macOS), set it immediately after creation and document the window.
- **FR-63** Never raise `SIGPIPE`.
- **FR-64** Retry `EINTR` internally, recomputing the remaining deadline.
- **FR-65** Worker CPU placement and connection steering (P2); NUMA-aware pools (P3).
**Drivers and outbound**
- **FR-66** One completion-shaped driver interface: operations own their buffers until completion; cancellation always ends in exactly one completion.
- **FR-67** Blocking driver: bounded worker pool, one connection per worker, an accept thread per listener with a wake channel, a bounded hand-over queue; refuse when full.
- **FR-68** Reactor driver: one event loop per worker (epoll, kqueue, IOCP); idle connections hold no read buffer.
- **FR-69** io_uring driver (P3): Linux only, non-default feature, automatic fallback to epoll.
- **FR-70** An executor adapter so worker threads park inside the reactor (P2, feature `executor`).
- **FR-71** `connect_host` resolves through `std` (never on a reactor worker) and tries addresses in order with per-attempt and total deadlines.
- **FR-72** Happy Eyeballs, simplified for `std` resolution (P2).
### 5.2 Non-functional
- **NFR-1** Zero runtime dependencies: `cargo tree -e normal` shows only `net-io`.
- **NFR-2** `unsafe` only under `src/sys/`; every block with a `// SAFETY:` comment naming the invariant and the documentation it relies on; a CI grep fails the build otherwise.
- **NFR-3** No allocation on hot paths after warm-up (accept, admission, read, dispatch, gate check, write, timers, completion post), proven by a counting allocator.
- **NFR-4** Zero CPU when idle: no periodic tick; measured over 60 s idle.
- **NFR-5** Memory bounded by configuration, never by peer behaviour.
- **NFR-6** No panics reachable from library code.
- **NFR-7** Descriptor hygiene: counts return to baseline after 100,000 connections across every close path.
- **NFR-8** Exactly one closed event with one `CloseReason` per connection on every path, including kill races and shutdown.
- **NFR-9** At least 20% faster than the best Rust competitor at equal or stronger security on each benchmark of section 9, p99 included, measured and published; a regression over 5% blocks a change.
- **NFR-10** Linux, macOS, Windows first class on stable and MSRV in CI; FreeBSD from 0.5.
- **NFR-11** Deterministic tests: the wheel, limits, and session state machine run under an injectable clock; `sys` has a fault-injection shim in test builds (D-045).
- **NFR-12** The gate check adds one predictable branch and one hook call per write system call; under 2 ns with a one-atomic gate [est.].
- **NFR-13** Fail closed on every hook error and every unverifiable inherited or handed-off descriptor.
- **NFR-14** Every public item documented with a runnable example; `docs/API.md` lists every item.
- **NFR-15** MSRV 1.85 (edition 2024); newer OS features through `sys` (D-031).
- **NFR-16** Minimum platforms (D-037): Linux 4.18, macOS 11, Windows 10 1809 / Server 2019, FreeBSD 13 for the full feature set; older systems get `Unsupported`, never undefined behaviour.
- **NFR-17** Soak: 24 hours at 0.7 and 72 hours before 1.0 with no memory, descriptor, or handle growth and no stuck connection.
- **NFR-18** Fewer system calls per request than each competitor on the echo benchmark; counts published.
- **NFR-19** A kill from another thread puts RST on the wire within 1 ms p99 on an idle system [est.].
- **NFR-20** `close_door` stops new accepts within 1 ms p99 [est.].
### 5.3 Enterprise and distributed (standard 2.16)
net-io holds per-process connection state (sockets, timers, the per-source table) and no state that must survive the process or be shared between nodes. The "stateful" row applies to what a deployment needs from it across processes and nodes:
| Point | How net-io meets it |
|---|---|
| correct under concurrency and load | `Send`/`Sync` checked at compile time; no global mutable state except two documented read-only-after-start facts; loom models for every concurrency primitive; soak tests (NFR-17) |
| observable | `EventSink` (zero cost unused), `Stats` snapshots, `CloseReason::as_str` stable names, `NetError::class` and stable variant names |
| operable | every limit configurable with a safe default and validated (FR-25); overload refuses cleanly (caps, `Busy`, `Shed`, `PostError::Full`); limits and capacity numbers documented in `docs/API.md` and, from 0.6, `PERFORMANCE.md` |
| secure | section 8; fail closed; the gate check cannot be skipped |
| many processes, many nodes | no process-local assumption in the API: hand-off moves listeners between processes; `SO_REUSEPORT` groups and systemd activation let several processes share a port; the control line lets a cluster controller shed or drain every node; `ConnectionId` is unique per process and paired with a node id by the application when it leaves the process |
| partitions and clock skew | each node protects itself locally (caps, deadlines, the gate's local state); timers use the monotonic clock only, so wall-clock skew between nodes cannot fire or delay a deadline |
| idempotent under retries | `close_door`, `open_door`, `drain`, `kill`, `stop` are idempotent; a second hand-off receive of the same blobs fails closed |
| long-lived systems | semantic versioning, MSRV policy, deprecation before removal, `docs/STABILITY-1.0.md` at 1.0 |
---
## 6. Architecture
### 6.1 Module map
```
src/
lib.rs crate docs, lints, Tier 1 functions, re-exports, VERSION
error.rs NetError, ErrorClass, CloseReason, RefusalCause, LimitKind, TimeoutKind, ConfigReason
addr.rs normalize, Cidr, PrefixKey, PrefixPolicy, TrustedProxies, Endpoint
id.rs ConnectionId, ListenerId, WorkerId, id blocks
info.rs ConnectionInfo, Transport, Origin, PeerCredentials, Lane
limits.rs Limits, LimitsBuilder, defaults, validation, MinRate, Pressure, SizeClass
opts/ ListenOptions, TcpOptions, Keepalive, Linger, SocketOptions, per-platform mapping
tcp.rs TcpListener, TcpConnection, ConnectOptions, connect_host, Happy Eyeballs (0.5)
udp.rs UdpSocket, amplification guard, source budgets, Datagram
local/ unix.rs (Unix + Windows AF_UNIX), pipe.rs (named pipes), credentials.rs
proxy/ v1.rs, v2.rs, tlv.rs, crc32c.rs, encode.rs, ProxyParser, ProxyHeader
gate.rs Gate, Admitted, Refusal, WriteCheck, Denial, AllowAll, KillHandle
tls.rs TlsAcceptor, TlsSession, TlsProgress, TlsEvent, TlsInfo, TlsError, NoTls
events.rs EventSink, Event, NoopSink
control.rs ControlLine, ControlState, NoControl, SimpleControl
session.rs Session, SessionFactory, SessionContext, Step, Phase, Input, Output
server/ builder.rs, server.rs, door.rs, drain.rs, adopt.rs, stats.rs, placement.rs
buf/ pool.rs (per-worker free lists per class), buffer.rs (Buffer, swap), wipe.rs
time/ wheel.rs (hierarchical timing wheel), clock.rs (Clock, MonotonicClock, ManualClock)
driver/ mod.rs (Driver, Operation, Completion, Token), blocking.rs, reactor/ (epoll, kqueue, iocp), uring.rs (P3), completer.rs
inherit.rs systemd, systemd_named, fastcgi_listener
handoff.rs ListenerHandle, manifest encode/decode, send/recv (Unix), share/receive (Windows)
sys/ the only unsafe: unix/, linux/, bsd/, windows/, shim.rs (fault injection, test builds)
```
`pub(crate)` by default; the public surface is exactly `docs/API.md`.
### 6.2 Connection lifecycle
```
┌──────────── refused (cap, gate, untrusted PROXY, hello refused) ─────────────┐
│ ▼
accepted ──▶ admitted ──▶ handshaking ──▶ open ⇄ parked ──▶ draining ──▶ closing ──▶ closed (one CloseReason)
(id, peer, (lane, (PROXY under (Session (offload (no new (flush,
caps) ticket) handshake phases: in flight, exchange) FIN or RST)
deadline, TLS) idle, park
receiving, deadline)
sending)
any state ──── kill / gate denial / limit / error / drain deadline ──────────────────────────▶ closing (abortive)
```
| State | Timers armed | Reads | Writes | Exits |
|---|---|---|---|---|
| accepted | none (microseconds) | no | refusal bytes only | admitted, refused |
| admitted | handshake | PROXY header and TLS records only | TLS records | handshaking, refused, closing |
| handshaking | handshake | yes (bounded by header caps) | TLS records | open, closing |
| open, Idle | idle | yes | no pending output | Receiving, closing |
| open, Receiving | read, rate in, exchange | yes | allowed | Sending, Idle, parked, closing |
| open, Sending | write, rate out, exchange | optional | yes | Idle, Receiving, closing |
| parked | park, exchange | no (bytes stay in the kernel) | no | open (wake), closing |
| draining | drain (server) plus the phase's timers | until the exchange ends | yes | closing |
| closing | graceful linger (2 s) or none (abortive) | drain only, bounded | flush only | closed |
Rules: no byte is read before `Gate::admit` succeeds; net-io moves Idle to Receiving itself on the first byte of an exchange; exactly one transition into `closed`, guarded by one atomic state word per connection (compare-and-swap on a closing flag), so a kill racing a FIN produces one reason.
### 6.3 The `sys` layer
- Each platform module declares only what net-io calls: `extern "C"` on Unix (the C library is already linked by `std`), `extern "system"` on Windows with `ws2_32` and `kernel32` (and `advapi32` for security descriptors, `bcrypt` for randomness). `ConnectEx` and `TransmitFile` are loaded at run time through `WSAIoctl(SIO_GET_EXTENSION_FUNCTION_POINTER)`, so `mswsock` is never linked (`AcceptEx` is not used: D-016).
- Constants and layouts are declared per target: `x86_64` and `aarch64` on `linux-gnu`, `linux-musl`, `apple-darwin`, `unknown-freebsd`, `pc-windows-msvc`, `pc-windows-gnu`. Other targets compile the blocking driver over `std` only and return `Unsupported` for what `std` lacks (D-014).
- Every struct has a `const` size and alignment assertion; every constant, size, field offset, and function signature is cross-checked in CI against `libc` and `windows-sys`, which are **dev-dependencies only** (`tests/layout.rs`).
- `sys` exposes only safe functions over owned or borrowed handles (`OwnedFd`, `OwnedSocket`, and their borrowed forms); no raw pointer or raw descriptor crosses the boundary.
- Windows: `WSAStartup(2.2)` once through `std::sync::Once` (reference counted, so it coexists with `std`'s own call).
- The full list of functions, constants, and structs per OS, with the documentation each relies on, is frozen in [`SYS-SURFACE.md`](./SYS-SURFACE.md). Adding to it needs a decision-log entry.
- Lints: the crate root becomes `#![deny(unsafe_code)]` and `#![deny(unsafe_op_in_unsafe_fn)]` in 0.3; only `src/sys/mod.rs` has `#![allow(unsafe_code)]`; a CI grep fails if the token `unsafe` appears outside `src/sys/` (doc comments excluded).
- Miri runs on every module outside `sys` with the shim; `sys` itself is covered by loopback tests on every OS, the layout cross-check, and sanitizer runs on Linux nightly.
**Fault-injection shim (D-045).** In builds with `cfg(test)` or the custom cfg `net_io_shim` (set by CI through `RUSTFLAGS="--cfg net_io_shim"` for the integration tests that need it), every `sys` function goes through a dispatch point a test can program ("the 3rd `send` on socket X returns 7 bytes", "the next `accept` returns `EMFILE`", "every `recv` returns `EINTR` once"). It is a cfg, not a Cargo feature, so it can never appear in the published feature list or be enabled by a dependent; normal builds compile no dispatch at all. `Cargo.toml` declares the cfg in `[lints.rust] unexpected_cfgs` so it does not warn.
### 6.4 Drivers
Completion-shaped (D-005): IOCP and io_uring report "this operation finished, here is your buffer"; epoll and kqueue report readiness and adapt by doing the non-blocking call themselves. The reverse (readiness on Windows) needs the undocumented `\Device\Afd` path that mio uses; completion keeps every backend on documented interfaces. The `Driver` trait is in `docs/API.md` (public Tier 3 from 0.5, D-029). `Completions` is a fixed array (256), so `wait` never allocates.
| | Blocking pool | Reactor | io_uring |
|---|---|---|---|
| Version | 0.3 | 0.5 | after 1.0, P3, feature `uring` (D-015) |
| Threads | `workers` (128, 256 KiB stacks) + one accept thread per listener | one per logical CPU, each with its own epoll, kqueue, or IOCP port | one per CPU, one ring each |
| `submit` | Unix: blocking `recv` with the remaining deadline as `SO_RCVTIMEO`; sends with `MSG_DONTWAIT`, waiting in `poll` on `EAGAIN`. Windows: non-blocking socket, `WSAPoll` until ready, then the call (D-052). The completion is queued locally | readiness: edge-triggered interest, call when ready; IOCP: `WSARecv`/`WSASend` (accept: one `WSAPoll` thread per listener, D-016) | an SQE |
| `wait` | local completions; when parked, the worker's parker until a completer wake or the park deadline | `epoll_wait` / `kevent` / `GetQueuedCompletionStatusEx` with the wheel's next expiry | `io_uring_enter` with a timeout |
| Kill wake | Unix: `shutdown(SHUT_RD)` wakes a blocked `recv`, the worker's wake pipe wakes a stalled send; Windows: the waiter's wake socket wakes `WSAPoll` (D-052, D-053), then abortive close by the worker | command queue + `DriverWaker` | `IORING_OP_ASYNC_CANCEL` |
Comparison [est.] (section 9 replaces it with measurements): the blocking pool needs 2 system calls per small exchange and has the lowest single-connection latency but falls behind beyond a few hundred active connections; the reactor needs about 3 (the wait amortized) and holds thousands of idle connections with state only; io_uring was measured about 10% ahead of epoll at 1,000 connections with CPU mitigations on and behind without them (Alibaba/OpenAnolis), and its security record (60% of the kernel exploits submitted to Google's kCTF VRP used it; Docker's default seccomp profile blocks it; `kernel.io_uring_disabled` exists) argues against making it a default.
**Accept distribution.** Linux: one listening socket in every worker's epoll with `EPOLLEXCLUSIVE` (4.5+); option at 0.6: one `SO_REUSEPORT` listener per worker, steered by `SO_INCOMING_CPU`. macOS, FreeBSD: one acceptor per listener hands sockets to the least-loaded worker's bounded inbox. Windows: one accept thread per listener waits in `WSAPoll` on the listener and a wake socket and accepts without `AcceptEx`, because a listener must never be associated with a completion port (D-016); it hands each socket to the least-loaded worker, which associates the accepted socket with its own port. Blocking: one accept thread per listener; a bounded hand-over queue (capacity = idle workers); full means refuse with `Busy`.
**IOCP idle connections.** A pending `WSARecv` with a real buffer pins it while the peer is silent; the reactor posts zero-byte receives for idle connections and reads into a pool buffer when one completes (one extra system call per wake-up, one buffer saved per idle connection; measured at 0.5). `FILE_SKIP_COMPLETION_PORT_ON_SUCCESS` is set unless `WSAEnumProtocols` shows a provider without `XP1_IFS_HANDLES`.
**Executor seam.** An executor's own driver trait (a worker that would park asks for ready events with a timeout) is implemented by an optional adapter over the reactor (feature `executor`, D-030), so one thread serves tasks and sockets with no extra hop.
### 6.5 Blocking driver
```
accept thread (per listener) workers (bounded pool, default 128)
poll([listener, wake], timeout) loop:
├─ wake: check door / control line take (conn, info) from the hand-over queue (blocks, zero CPU)
├─ readable: accept up to accept_batch session.on_open
│ reserve-descriptor routine on EMFILE loop on Step:
│ caps (global, listener, source) Read ─▶ recv(timeout = earliest of read/idle/exchange/rate check)
│ Gate::admit (no read yet) Write ─▶ per chunk: control check, Gate::check, send
│ hand-over queue try_push Park ─▶ parker.wait(park deadline) until the Completer posts
│ full ─▶ refuse (Busy), RST Close ─▶ graceful or abortive close, one closed event
└─ repeat return to the pool
```
- `poll` / `WSAPoll` on the listener plus a wake socket lets `close_door`, `shutdown`, and the control line interrupt the accept thread without closing the listener, which hand-off needs.
- Unix reads: blocking `recv` with `SO_RCVTIMEO`, set only when the target moves by more than one tick, so most reads are one system call. Unix sends: `MSG_DONTWAIT` first (usually one call); on `EAGAIN`, `poll` on the socket and the worker's wake pipe with the write deadline.
- Windows: Microsoft documents a socket whose blocking receive or send timed out as being in an indeterminate state. Spike S-2 saw no data lost after a timeout, but a lost or reordered byte in the middle of a stream would desynchronize the protocol framing, which is how request smuggling starts, so net-io does not rely on it: Windows sockets in the blocking driver are non-blocking, every wait is `WSAPoll` on the socket and the waiter's wake socket with the deadline, then the call. Cost: two system calls per read where Unix needs one [est. under 1 µs extra]; the production path on Windows is the IOCP reactor (D-052).
- The minimum rate on the blocking driver: the read timeout is capped at the rate-check interval (1 s) and the rate is checked on every return from `recv`.
- An idle keep-alive connection occupies a worker, so the default idle deadline is 5 s and `pressure()` reports the fraction of busy workers (D-028).
### 6.6 Buffers and memory
- **Size classes** 4, 16, 64 KiB by default (powers of two from 1 KiB to 1 MiB). Each worker owns a free list per class; buffers are allocated on first need and never freed while the server runs.
- **Budget:** the sum of all buffers ever allocated is bounded by `buffer_budget` (validated at least `workers x 64 KiB x 2`); when spent, `take` returns `None`, the connection stops reading (back-pressure through the kernel's receive window), and `LimitHit { BufferBudget }` is emitted at most once per second per worker.
- **Asynchronous drivers:** a buffer given to `Operation::Receive` belongs to the driver until its completion is reaped, even when the connection is killed; the kill path cancels and waits for the completion before returning the buffer (6.10).
- **Zero copy:** `on_data` receives `&mut Input`, a view into the connection's buffer; compaction happens only when the free tail is under a quarter (one `copy_within`, amortized).
- **Swap** (`Input::swap`): O(1) exchange with an empty caller-owned buffer of the same class; the bytes never move. For request bodies and WebSocket messages handed to another owner; the request head is copied by the protocol layer instead (cross-crate decision X-3).
- **Wipe on release** (D-027): volatile zeroing of the used prefix; 50 to 100 ns per 2 KiB [est.].
- **Output:** a bounded chunk queue (pool buffers, `&'static [u8]`, file ranges); 64 KiB of pool memory per connection; vectored writes of up to 64 chunks per call (below `IOV_MAX`).
### 6.7 Accept path and accept storms
Per wake-up, per listener, until `accept_batch` or `EAGAIN`: (1) control line check (one atomic load; door closed, `Draining`, or `Stopping` stops the batch and leaves the rest in the backlog); (2) `accept4` / `accept` with the FR-10 error classes; (3) caps, refusing with RST before any read; (4) id from the worker's block, normalized peer, `ConnectionInfo`; (5) `Gate::admit` (a panic counts as refusal); (6) per-connection options, only where the OS does not inherit them from the listener (spike S-4); (7) register with the driver and arm the handshake or idle deadline.
**Reserve descriptor.** Each Unix acceptor holds one spare descriptor (`/dev/null`). On `EMFILE`/`ENFILE`: close the spare, accept one connection, close it with RST, reopen the spare, emit `DescriptorPressure`, pause with backoff (1 ms doubling to 1 s, reset after one good batch, no CPU while paused). Best effort: another thread can take the freed slot (the caveat Twisted documents for the same technique); the backoff then applies. Windows socket handles do not come from a small per-process table; there the pause alone handles `WSAEMFILE` and `WSAENOBUFS`.
**Backlog.** Default 1,024; Linux caps it silently at `net.core.somaxconn` (4,096 since 5.4, 128 before), so net-io reads `/proc/sys/net/core/somaxconn` once and reports the effective value. SYN floods are the kernel's (SYN cookies), documented in `PLATFORM-NOTES.md`.
### 6.8 Limits
Defaults and the reason for each are in `docs/API.md` ("Defaults and reasons"). Design notes:
- **Per-source table** (D-025): open addressing over `(PrefixKey, count)` with capacity the next power of two above `2 x max_connections`, so live entries never exceed `max_connections` and the table never fills; backward-shift deletion, no tombstones; keyed hash with a per-process random key read once from the OS (no attacker-chosen collisions). On the reactor the table is shared and striped with compare-and-swap counts [measured at 0.5 against per-worker shards]. The 0.3 blocking driver uses 16 shards of std maps keyed by std's `RandomState` instead (D-062).
- **Minimum rate:** per direction, a ring of ten one-second counters; a wheel timer every second only while the direction is active.
- **Adaptive deadlines** (P2, 0.6): above `pressure.high` of `max_connections`, handshake and idle deadlines of new and idle connections are multiplied by `pressure.factor` (floor 1 s), restored below `pressure.low`.
- **Lane reservations** (P2, 0.6): slots reserved for lanes out of `max_connections`; a lane may use the shared slots or the reserve of any less important lane; a flood of unimportant connections cannot take the last descriptors needed for health checks.
### 6.9 Timers
One hierarchical timing wheel per worker (Varghese and Lauck, 1987): 6 levels of 64 slots, 1 ms tick (1 to 100 ms configurable), about 2.2 years at 1 ms, an overflow list beyond. Intrusive slots per connection (handshake, idle, read, write, exchange, park, rate), so insert and cancel allocate nothing. Lazy extension: a read that extends a deadline only stores the new deadline; the entry is re-inserted when it fires early. Never early; tickless (the driver's wait timeout is the next occupied slot, found with a 64-bit occupancy bitmap per level); saturating `u64` tick arithmetic; a forward jump fires every expired slot once, in order, at most 1,024 per iteration.
The wheel ships with the reactor in 0.5 (D-063). The blocking driver holds one connection per worker and computes its few deadlines from the connection's progress time; `SO_RCVTIMEO` (Unix) or the `WSAPoll` timeout (Windows) carries the nearest one.
### 6.10 Cancellation and kill
| Driver | Kill sequence |
|---|---|
| Blocking | compare-and-swap the closing flag; then wake the worker: Unix `shutdown(SHUT_RD)` on the connection (a blocked `recv` returns 0 at once, spike S-3; the worker checks the flag before treating 0 as end of stream) and one byte to the worker's wake pipe (a send waiting in `poll`); Windows one datagram to the waiter's wake socket (`WSAPoll` returns). The worker sees the flag, sets linger 0, closes, emits the event. The killing thread never closes the descriptor itself, and it touches the descriptor or wake channel only under the kill guard (D-053). |
| Reactor (epoll, kqueue) | push `Kill(id, reason)` on the owning worker's bounded command queue and wake it; the worker deregisters, sets linger 0, closes, emits. |
| IOCP | `CancelIoEx(socket, NULL)`, buffers stay owned, close only after **every** outstanding completion was dequeued (Microsoft documents three outcomes for a cancelled operation; the `OVERLAPPED` and buffer must live until it completes). |
| io_uring | `IORING_OP_ASYNC_CANCEL` by user data; buffers return after the original CQE and, for zero-copy sends, after the notification CQE. |
A `KillHandle` holds the worker index, the `ConnectionId`, a generation number, and the worker's command queue; a generation mismatch returns `false`.
**Kill guard (D-053).** A killer that makes a system call on the connection's socket or on a wake channel must never hit a descriptor the owner already closed and the OS reused. The connection's state word carries a `KILLING` bit: the killer sets it with the closing flag in one compare-and-swap (only if the connection is not already closed), makes its wake call, then clears it; the owner's close path waits until `KILLING` is clear before it closes the socket or unregisters its wake channel. The wait is bounded by one system call in the killer. Model-checked with loom (kill vs close vs FIN vs drain).
### 6.11 Gate seam (bouncer-io)
The shape agreed for both crates' 0.2 design locks (D-018; bouncer-io's open decision O-6, option (a)):
```rust
pub trait Gate: Send + Sync + 'static {
type Ticket: Send + 'static;
fn admit(&self, info: &ConnectionInfo, stage: AdmitStage) -> Result<Admitted<Self::Ticket>, Refusal>;
fn readmit(&self, ticket: &mut Self::Ticket, info: &ConnectionInfo) -> Result<(), Refusal>;
fn check(&self, ticket: &mut Self::Ticket, write: &WriteCheck) -> Result<(), Denial>;
fn closed(&self, ticket: Self::Ticket, info: &ConnectionInfo, reason: CloseReason) { let _ = (ticket, info, reason); }
}
```
- The ticket is stored beside the connection and passed back on every call, so the check before a write is one load through the gate's own state (bouncer: one atomic epoch load), with no lookup by id.
- One trait instead of separate admission and egress traits: the same per-connection ticket serves both, and a server sets it with one call (`.gate(bouncer)`).
- `closed` gives the ticket back with the one close reason: the gate releases its counts and can turn `SlowPeer`, `ProxyInvalid`, `ProxyUntrusted`, `TlsFailed`, `HelloRefused`, and timeouts into strikes without parsing events.
- Denials carry a code; the connection's `Session` writes the protocol's denial for it, because only the session knows its framing state. The denial is written only on a clean response boundary (`Output::begin_response`); otherwise the connection is reset with no denial bytes (D-032).
- `GateTag` (D-051): the gate's opaque name for the connection, stored in `ConnectionInfo::gate` for protocol engines, and an optional per-response tag the session sets with `Output::begin_tagged_response`, passed to `check` as `WriteCheck::response`.
- With `AllowAll` (`Ticket = ()`, `GateTag(0)`), the checks compile to nothing.
What the transport cannot do, stated plainly: bytes already handed to the kernel cannot be recalled. Responses that must be all-or-nothing are pushed in full before they are written.
### 6.12 Control line
net-io's view of an application-wide urgent signal: one `Acquire` load of the epoch at the top of every worker loop, before every accept batch, and before every admission; the state is read only when the epoch moved. Reactions: `Shed { from_lane }` refuses new connections in that lane and every less important one and reports `pressure() == 1.0`; `Lockdown` refuses new connections on listeners not marked `lockdown_allowed`; `Draining` closes the door and drains; `Stopping` kills everything left with `Shutdown`. `ServerHandle::wake_all` reaches workers asleep in the kernel (eventfd, `EVFILT_USER`, `PostQueuedCompletionStatus`, the blocking pool's wake socket and parkers).
### 6.13 Completer
A bounded multi-producer single-consumer ring per worker (Vyukov-style, power-of-two capacity, a sequence number per slot), loom-tested. `post` on a full ring returns `Full`; a completion for a connection that closed meanwhile is dropped and counted (generation check); the poster wakes the worker only on the empty-to-non-empty transition. The value is a `u64`, so nothing is allocated or dropped.
### 6.14 PROXY protocol
Modes `Off` and `Required { V1 | V2 | Both }`; no optional mode (the specification forbids guessing). Trust is checked at accept from the socket address, before any read. The parser is a sans-IO state machine with a resume offset, borrowing TLV values from the caller's buffer; `to_facts` copies ALPN (32 bytes), authority (255), unique id (128), and SSL flags into fixed arrays inside `ConnectionInfo`. The v2 address-and-TLV cap is 1,024 bytes by default, configurable to 65,519 (D-040). After the parse: the real peer is normalized, the per-source cap moves to it, `Gate::readmit` runs. Strictness rules for v1 and v2 are in `docs/API.md`.
### 6.15 TLS seam
In place (D-019, cross-crate decision X-2): the engine's `step` decrypts in the connection's read buffer and returns plaintext as a range of it; `encrypt` writes records straight into net-io's output buffer. The condition X-2 set (rustls's unbuffered API reassembles a handshake message that spans several records inside the caller's buffer) holds: rustls 0.23.46's unbuffered path calls the handshake deframer's `coalesce`, which moves the fragments together in the caller's buffer (`src/conn/unbuffered.rs` -> `ConnectionCommon::deframe` in `src/conn.rs` -> `HandshakeDeframer::coalesce` in `src/msgs/deframer/handshake.rs`). Consequence: the input buffer must hold the largest handshake message being assembled, which the session reports through `input_capacity_required` and net-io meets by growing the read buffer up to `read_class_max`.
The ClientHello is parsed by the engine inside `step` from net-io's buffer (no copy); fingerprinting and refusal are the engine's hello hook (tls-net calls bouncer-io), surfaced to net-io as `TlsFailure::HelloRefused`. net-io's earlier plan to copy the hello into a per-worker scratch buffer and call its own hook is dropped (D-019): the engine already holds the parsed hello, and a second hook would mean a second parse.
Order: accept, admit, PROXY, readmit, `TlsAcceptor::session` (capacity refusal before reading), handshake under the handshake deadline, `info()` into `ConnectionInfo`, protocol chosen by ALPN. Outbound: gate check on plaintext before `encrypt` and before each ciphertext flush; denial encrypted, then `close_notify`, then reset.
### 6.16 Drain, door, shutdown, hand-off
```
BOOT SHUTDOWN RESTART
build(): bind or adopt listeners, close_door() (first) the supervisor or the old process
start workers, door CLOSED drain(deadline) holds the listening socket
[drop privileges] idle: close now new process: build() with listeners
open_door() (last) active: finish the exchange received by hand-off ─▶ open_door()
deadline: kill (DrainDeadline) old process: close_door()
shutdown(): close listeners, ─▶ drain(deadline) ─▶ exit
join workers the listening socket is never closed
```
- `close_door` sets the door word and wakes every acceptor (NFR-20). Connections in the kernel backlog stay there: the next process accepts them in a restart; `shutdown` resets them in a final stop.
- Overlap during a restart is allowed (D-042): both processes may accept for a moment; each connection belongs to one.
- **Unix manifest:** one `sendmsg`, up to 64 descriptors (`SCM_RIGHTS`; Linux's per-message limit is 253), body: magic `NIOH`, version 1, count, per listener kind, name (up to 64 bytes), local address, options digest, then CRC32C. The receiver uses `MSG_CMSG_CLOEXEC` where available, fails on `MSG_CTRUNC`, checks count, verifies each descriptor (`SO_TYPE`, `SO_ACCEPTCONN`, `getsockname` equals the manifest). Any failure closes all received descriptors (fail closed). The channel is a Unix socket in a `0700` directory; the receiver checks the sender's uid (default its own) and, if configured, pid.
- **Windows:** `WSADuplicateSocketW(listener, target_pid)` yields a `WSAPROTOCOL_INFOW` for one target process, sent over an owner-only named pipe; the receiver checks `GetNamedPipeClientProcessId` and calls `WSASocketW(FROM_PROTOCOL_INFO, WSA_FLAG_OVERLAPPED)`. Windows does not make the blob single use (spike S-1: a second use returns the same handle, a use after the sender closed returns a dead one), so net-io does: the manifest carries a 128-bit random nonce, `receive_shared` refuses a nonce it already accepted in this process, `SharedListeners::write_to` consumes the value so it cannot be sent twice, and every received socket is verified before use (D-054). Listeners are never associated with a completion port in any process, so a received listener works in the new process exactly as in the old one (D-016).
### 6.17 Inheritance
- **systemd** (FR-42): `LISTEN_PID` must equal this process (otherwise an empty list), `LISTEN_FDS` at most 64, `LISTEN_FDNAMES` optional, `LISTEN_PIDFDID` checked against the inode of this process's pidfd on Linux when present (systemd 259+). Descriptors from 3, each verified (`fstat` socket, `SO_ACCEPTCONN`, `SO_DOMAIN`, `SO_TYPE`) and set close-on-exec. The variables are not unset (D-036).
- **FastCGI** (FR-43): descriptor 0 must be a socket, listening, with `getpeername` failing with `ENOTCONN`; adopted as TCP or Unix by `SO_DOMAIN`. Windows: OD-26.
### 6.18 Outbound connections
`connect`: non-blocking connect, wait for writability within the deadline (blocking driver: `poll` on Unix, `WSAPoll` on Windows, D-056), then `SO_ERROR`. The Windows reactor uses `ConnectEx` (needs a bound socket). `connect_host`: `std` resolution off the worker, addresses in order, per-attempt deadline = remaining / remaining addresses (at least 1 s). Happy Eyeballs at 0.5 (D-044).
### 6.19 Placement (P2) and NUMA (P3)
`Placement::Off` (default), `PinPerCore` (physical cores before SMT siblings), `Explicit(cpus)`. Linux `sched_setaffinity` plus `SO_INCOMING_CPU` per-worker reuse-port listeners; Windows `SetThreadGroupAffinity` (processor groups above 64 CPUs); FreeBSD `cpuset_setaffinity`; macOS: `Unsupported` (D-037). NUMA (P3): one pool per node, first touch from the pinned thread.
### 6.20 Fork safety and signals
Every descriptor and handle is close-on-exec / non-inheritable; on macOS `socket` and `accept` have no atomic flag, so `FD_CLOEXEC` is set immediately after creation (a `fork` plus `exec` by another thread in that window can leak one descriptor; documented with the advice to start children before the server or use `posix_spawn` with `POSIX_SPAWN_CLOEXEC_DEFAULT`). net-io never forks; a `Server` records its creating pid and its cold calls return `ForkedChild` in a child. No signal handling; `SIGPIPE` is avoided on every write because a library cannot rely on the host's disposition. `EINTR` is retried with the remaining deadline.
### 6.21 Events, stats, ids
`EventSink` is a type parameter of the builder (D-033); `NoopSink` compiles away; events are `Copy`, at most 64 bytes, borrow nothing; `Refused` and `LimitHit` coalesce above 1,000 per second per worker. Stats are per-worker plain counters published as relaxed snapshots once per loop. `ConnectionId`s come from per-worker blocks of 1,024 taken with one `fetch_add` on a process-wide `AtomicU64`; a small block-to-worker map lets `ServerHandle::kill` find the owner.
---
## 7. Deployment modes
Our server never runs inside Apache or nginx. It runs beside them, in one of these modes. All of them hand the upper layers the same normalized connection, so nothing above the ingress adapter knows which mode is active.
| Mode | How | Who handles it |
|---|---|---|
| **Direct** | binds :80/:443 itself, TLS via tls-net | net-io + tls-net |
| **Reverse proxy, TCP** | Apache `ProxyPass` / nginx `proxy_pass` to `127.0.0.1:port`; real client via PROXY protocol or `X-Forwarded-For` from a trusted proxy | net-io (+ the web layer for headers, with the same `TrustedProxies`) |
| **Reverse proxy, Unix socket** | same, over a Unix socket; no port exposed | net-io local sockets |
| **FastCGI** | Apache `mod_fcgid` / `mod_proxy_fcgi`, nginx `fastcgi_pass` | fcgi-net (sans-IO) over net-io; descriptor 0 adoption (FR-43) |
| **CGI** | the web server runs our binary once per request | web layer adapter over stdin/stdout, last resort |
| **Socket activation** | systemd passes the listening sockets | `net_io::systemd()` |
**cPanel note.** On a cPanel server with WHM/root (VPS or dedicated), a long-running daemon behind Apache `ProxyPass` (or EA-Nginx) is fine. On shared cPanel hosting, a process that keeps a port open is usually not allowed and FastCGI is often the only real option (open decision O-2).
---
## 8. Security
### 8.1 Threat model
net-io is the first and last code every connection touches. Attackers range from scanners to botnets with residential proxies. net-io must stop, on its own, the attacks that are transport problems (every layer stands alone), and it must make the gate's enforcement impossible to skip. Assets: descriptors, memory, worker threads, CPU, the integrity of the client address, the guarantee that no byte leaves a revoked connection, listening sockets during restarts, local IPC endpoints.
Owned elsewhere: bans and rates (bouncer-io), content (customs-sec), identity (gates), floods larger than the link (upstream), SYN floods (kernel SYN cookies; documented).
### 8.2 Threats, defences, tests, costs
Every threat has an attack test that must fail to get through. Costs are [est.] until measured.
| # | Threat | Defence | Test | Cost |
|---|---|---|---|---|
| T-1 | connection flood exhausting descriptors | caps before any read; gate before any read | `test_accept_flood_at_cap_refuses_with_rst` | one table update per accept, under 50 ns |
| T-2 | descriptor exhaustion making accept spin or stall | reserve descriptor, accept backoff | shim `EMFILE`; real low `RLIMIT_NOFILE`; CPU near zero, backlog drains when descriptors return | one spare descriptor per acceptor |
| T-3 | slowloris | handshake and read deadlines, inbound minimum rate, exchange deadline | one byte every 900 ms; closed `SlowPeer { In }` by grace + window | one timer per active connection per second |
| T-4 | slow-read | write deadline, outbound minimum rate, output budget | zero receive window; closed `SlowPeer { Out }` or `WriteTimeout`; memory never above the budget | same |
| T-5 | idle keep-alive hoarding | idle deadline; adaptive deadlines (P2); blocking idle 5 s | 10,000 idle at 90% of cap; new clients admitted | none below pressure |
| T-6 | handshake that never finishes | one handshake deadline for PROXY and TLS | half a PROXY header, half a ClientHello | one timer |
| T-7 | memory exhaustion through many connections | idle connections hold no read buffer; budgets; back-pressure | 50,000 connections each sending 1 byte; RSS bounded | none |
| T-8 | oversized PROXY header | 107-byte v1 cap; v2 length checked before the body | v2 declaring 65,535 bytes closed at once | none |
| T-9 | spoofed client address via PROXY from an untrusted peer | trust checked at accept; untrusted peer on a Required listener closed before reading | valid header from a non-trusted address: `ProxyUntrusted` | one set lookup |
| T-10 | missing PROXY header behind a proxy | `Required` fails closed; no optional mode | trusted peer sends `GET /`: `ProxyMissing` | none |
| T-11 | PROXY header injected by a client through a proxy that does not strip it | exactly one header is read; a second is application data | second line never interpreted as PROXY | none |
| T-12 | PROXY v2 TLV bugs | bounds-checked walk, checked arithmetic, fuzzing, property tests | fuzz target `proxy_v2`; property: never panics, never reads past the length | none |
| T-13 | framing someone else through a forged address | as T-9; `client_key` uses `real_peer` only from a trusted header | as T-9 plus the gate's own test | none |
| T-14 | data leaving a revoked connection | gate check before every write system call; wipe; abortive close | revoke between two chunks; no byte of chunk 2 on the wire (loopback capture) | one hook call per write, under 2 ns |
| T-15 | partial response leaking before a denial | denial only on a clean boundary | revoke after the first byte of a streamed response: reset, no denial bytes mixed in | none |
| T-16 | residual request data in reused buffers | wipe on release (default on) | secret read on connection A is zeroes for connection B | 50-100 ns per 2 KiB |
| T-17 | kill races (double close, reused descriptor) | closing flag CAS; only the owner closes; generations | loom model kill vs FIN vs drain; 1 million racing kills | one CAS per close |
| T-18 | IOCP / io_uring use-after-free on cancellation | buffers owned until the completion is reaped | kill during pending `WSARecv` with a guard-page allocator; ASan for io_uring | none |
| T-19 | accept storm starving established connections | accept batch; `EPOLLEXCLUSIVE` | 50,000 connects per second while 1,000 echo; their p99 within 2x | none |
| T-20 | offload flood filling completion queues | bounded ring, `Full`, park deadline | 1 million posts to one worker; never above capacity | none |
| T-21 | event sink as a bottleneck | coalescing above 1,000 per second per worker | refusal flood; sink sees about 1,000 per second plus counts | one counter |
| T-22 | hash flooding the per-source table | keyed hash, random per-process key | addresses chosen for an unkeyed hash make no long chains | about 2 ns per hash |
| T-23 | Unix socket hijack or eavesdropping | default `0600`, private directory guidance, `bind_atomic`, `require_uid` | another uid connects to a `0600` socket and is refused | one `getsockopt` per accept |
| T-24 | stale-path deletion of a non-socket (symlink attack) | `lstat`, remove only a refusing socket | path is a symlink to an important file: `StaleSocketInUse`, target untouched | one `lstat` |
| T-25 | Linux abstract socket squatting | abstract names only with `require_uid` | another uid binds first; ours fails; a client of another uid refused | one `getsockopt` |
| T-26 | named-pipe squatting | `FILE_FLAG_FIRST_PIPE_INSTANCE`; owner-only descriptor | another process creates the pipe first: `PipeNameInUse` | none |
| T-27 | pipe client impersonated by a malicious server | `SECURITY_SQOS_PRESENT \| SECURITY_IDENTIFICATION` | test server impersonates: identification level only | none |
| T-28 | remote access to a local pipe | `PIPE_REJECT_REMOTE_CLIENTS` always | SMB connect when a lab exists; flag inspection otherwise | none |
| T-29 | hand-off channel abuse | `0700` directory, peer uid/pid, magic, version, CRC32C, per-descriptor verification, all or nothing | manifest from another uid; a regular file descriptor; truncated control data: all rejected, every descriptor closed | under 1 ms per hand-off |
| T-30 | Windows hand-off to the wrong process, or a replayed hand-off | `WSADuplicateSocketW` targets one pid; pipe client pid checked; net-io enforces single use with a manifest nonce (Windows does not, spike S-1); every received socket verified | replay the blob in another process: `WSASocketW` fails; replay it in the target process: refused by the nonce check | one table lookup per hand-off |
| T-31 | descriptor leak into child processes | close-on-exec everywhere | spawn a child and list its descriptors / handles | none (macOS window documented) |
| T-32 | `SIGPIPE` killing the process | `MSG_NOSIGNAL`, `SO_NOSIGPIPE` | a C host with default disposition writes to a reset connection and survives | none |
| T-33 | systemd variables meant for another process | `LISTEN_PID`, `LISTEN_PIDFDID`, per-descriptor checks | a child with inherited variables gets an empty list | none |
| T-34 | descriptor 0 not a listening socket under FastCGI | `SO_ACCEPTCONN` and `ENOTCONN` checks | terminal, file, connected socket: `None` | 3 system calls once |
| T-35 | UDP amplification | no-amplification guard (factor 1) until proof | small request from a spoofed source: replies never larger | under 30 ns per send |
| T-36 | UDP source flood and table exhaustion | per-source budgets, bounded table with eviction, shared overflow bucket | 1 million sources; memory bounded; verified peers keep budgets | under 30 ns per datagram |
| T-37 | truncated datagram treated as complete | truncation always reported | 70 KB into a 1 KB buffer: `truncated` | none |
| T-38 | port hijacking on Windows | `SO_EXCLUSIVEADDRUSE` by default | second process binds with `SO_REUSEADDR`: fails | none |
| T-39 | privileged port held while running as root | bind, then drop privileges, then `open_door` | integration test in shield-sec; net-io: `open_door` is separate | none |
| T-40 | `Server` used after `fork` | creator pid; `ForkedChild` | fork and call `open_door` in the child | one comparison |
| T-41 | io_uring kernel attack surface | off by default, feature-gated, automatic fallback | under Docker's default seccomp: epoll and a `DriverFallback` event | none |
| T-42 | timer storms caused by peers | lazy extension, intrusive slots, bounded expirations | 100,000 connections extending every millisecond; CPU per connection flat | none |
| T-43 | hello inspection bypassed by fragmenting the ClientHello | the engine reassembles within its bound; net-io's read buffer limit and handshake deadline bound it too | ClientHello in 1-byte records | bounded |
| T-44 | refusal path used for amplification | at most 512 bytes, one non-blocking attempt, TCP after the handshake only | SYN flood triggers no writes; completed connections get at most 512 bytes | one system call |
| T-45 | gate or session panic letting traffic through | a gate panic is refuse / deny; a session panic closes the connection | test hooks that panic | none |
### 8.3 What net-io does not cover
- Bytes already handed to the kernel cannot be recalled; all-or-nothing responses must be complete before they are written.
- SYN floods are the kernel's (SYN cookies); net-io cannot see half-open connections.
- Floods larger than the link need upstream help.
- A compromised trusted proxy can claim any client address; keep the list minimal.
- macOS: a descriptor can leak to a child created by another thread between `socket` / `accept` and `fcntl(FD_CLOEXEC)`.
- Policy (who is banned, which rate) is the gate's; net-io's per-source cap is a concurrency cap only.
### 8.4 Defence ideas per version (hypothesis, method, cost)
| Version | Idea | Hypothesis | Method | Expected cost |
|---|---|---|---|---|
| 0.3 | reserve descriptor plus accept backoff | descriptor exhaustion no longer makes the accept loop spin or stall | shim-injected `EMFILE`, real low `RLIMIT_NOFILE`, CPU measured during pressure | one descriptor per acceptor |
| 0.4 | hand-off manifest with CRC32C, peer check, per-descriptor verification | a local attacker cannot inject or steal listeners during a restart | hostile sender tests (other uid, wrong descriptor types, truncation) | under 1 ms per restart |
| 0.5 | zero-byte receives on IOCP for idle connections | memory under an idle-connection flood on Windows stays at state-only levels | 10,000 idle connections, RSS against 4 KiB posted receives | one extra system call per wake-up |
| 0.6 | adaptive deadlines under pressure | hoarding near the cap is cleared four times faster without hurting normal clients | load test at 90% of cap with 10% slow clients | none below the threshold |
| 0.7 | deterministic fault injection across the whole state machine | every partial-IO and error path is exercised on every OS | shim schedules covering every system call site, in CI | test time only |
| 0.8 | lane reservations at the connection level | health checks and admin access survive a flood of unimportant connections | flood with low-importance connections; important connects still succeed | one counter per lane |
| 0.9 | per-source handshake-failure counts handed to the gate as strikes (net-io counts, the gate decides) | scanners that fail PROXY or TLS repeatedly are banned within seconds | scanner simulation with a gate; `Gate::closed` carries the reasons | one counter per failure |
| 1.0 | `SO_ATTACH_REUSEPORT_CBPF` seam for kernel-side steering and drops | known-bad prefixes can be dropped before accept on Linux | prototype with the gate's prefix export; accept-path CPU under a flood | kernel side only |
| 0.2 (new) | ticket returned to the gate at close (`Gate::closed`) | the gate learns every transport-level misbehaviour (slow peer, bad PROXY, TLS failure) with its connection's ticket, so strikes need no event parsing and no lookup | bouncer-io integration test: a slowloris connection produces a strike on its ticket's keys | one hook call per close |
---
## 9. Performance
### 9.1 Targets
All targets are [est.] until the 0.6 measurement. "Best competitor" is the fastest Rust library on that benchmark at equal or stronger security (security-sensitive benchmarks give competitors equivalent limits in user code; the harness documents each configuration).
| Id | Benchmark | Metric | Target |
|---|---|---|---|
| B-1 | accept rate, 1 listener, connect-close clients | connections per second, p99 accept-to-admission | at least 1.2x the best competitor |
| B-2 | echo, 64 B, 1 connection | round trips per second, p50, p99 | at least 1.2x (blocking vs `std::net` threads; reactor vs tokio) |
| B-3 | echo, 64 B, 100 / 1,000 / 10,000 connections | throughput, p99, p99.9 | at least 1.2x tokio; within 10% of monoio/glommio on Linux at 0.6, at least equal by 1.0 |
| B-4 | echo, 16 KiB, 1,000 connections | bytes per second, CPU per GB | at least 1.2x |
| B-5 | idle connections | RSS per idle connection attributable to the library | under 512 B, at least 1.2x lower than tokio |
| B-6 | 100,000 idle (Linux), 10,000 (all OSes) | RSS, idle CPU | bounded by configuration; idle CPU near zero |
| B-7 | gate check overhead | ns per write, `AllowAll` and a one-atomic gate | under 1 ns and under 2 ns |
| B-8 | timer insert / cancel / lazy extend | ns per operation | under 20, under 20, under 2 |
| B-9 | PROXY v1 TCP4, v2 IPv4 without TLVs, v2 with 5 TLVs and CRC32C | ns per header | under 100, under 50, under 300 |
| B-10 | completer post-to-wake | µs from `post` to `on_wake` on an idle worker | under 10 µs p99 |
| B-11 | kill latency | µs from `kill` to RST on the wire | under 1 ms p99 |
| B-12 | drain of 10,000 idle connections | ms to `DrainReport` | under 100 ms |
| B-13 | system calls per echo round trip | count | at most 3 (reactor), 2 (blocking, Unix), 3 (blocking, Windows: D-052) |
| B-14 | allocations per connection after warm-up | count | 0 |
### 9.2 Competitors and harness
Rust: `std::net` thread per connection, `mio`, `tokio`, `monoio` and `glommio` (Linux, io_uring), `compio`. libuv and Seastar for context only. An unpublished `compare/` crate with its own `[workspace]` holds one echo and one accept server per competitor and an in-house open-loop load generator (coordinated-omission-free timing as in `wrk2`). criterion microbenchmarks in `benches/` for B-7 to B-10 and the pool. Reference hardware: Ryzen 9 9950X3D, Windows 11 and WSL2 Ubuntu; published Linux numbers from bare metal or labelled "WSL2". Method: fixed governor, disjoint pinned cores, 10 runs, median and spread, system calls per operation, allocations, RSS slope; every number names its commit, hardware, OS, kernel, and command line. Load benchmarks run before each release from 0.6; a regression over 5% blocks the release.
### 9.3 Where the speed comes from (hypotheses for 0.6)
1. No runtime layer: sessions are called directly by the worker loop, with no future, waker, or task queue between.
2. No allocation per connection or per IO after warm-up.
3. Lazy timer extension: no wheel operation per read.
4. Fewer system calls: `accept4` with flags, batched accepts, `SO_RCVTIMEO` only on change, vectored writes, one wake per completion burst.
5. Idle connections hold no buffers, so caches stay warm for active ones.
6. The gate, the sink, and the session factory are monomorphized; no-op versions compile away.
7. The gate ticket removes a per-write lookup (D-018).
If a hypothesis fails, the roadmap records it and the design changes; the 1.2x gate is not waived without James in writing.
---
## 10. Ecosystem and dependencies
### 10.1 net-io's own dependencies
- **Runtime:** none, in every feature combination (NFR-1, D-014).
- **Dev only:** `criterion`, `proptest`, `loom` (under `cfg(loom)`), and `libc` / `windows-sys` for the layout cross-check only. Competitor crates live in the unpublished `compare/` crate, never in net-io's graph.
- **REPS exception (D-047):** REPS says "MUST use tokio for async runtime". net-io uses no async runtime (D-004); the portfolio standard records that clause as a known mistake, and this plan records net-io's exception. Whether REPS itself changes portfolio-wide is James's open decision O-1.
### 10.2 Existing crates the stack reuses
| Crate | Role in the stack |
|---|---|
| `http-net` | HTTP/1.1 engine, sans-IO, no dependencies (0.6 published) |
| `parse-net` | shared grammar for the protocol crates |
| `ws-net` | WebSocket engine, sans-IO |
| `wire-codec` | binary framing for non-HTTP protocols |
| `rpc-io` | JSON-RPC 2.0 engine |
| `clock-lib` | monotonic time; an application can implement net-io's `Clock` over it in one line (D-038) |
| `crypt-io`, `key-vault` | cryptography and key storage (gate-net, sessions, tls-net) |
| `log-io`, `metrics-lib` | logging and metrics (implement `EventSink`) |
| `signal-mod` | OS signals, turned into `ServerHandle::request_stop` or the control line by the application |
| `sys-lib` | process guard, open-file limit, boot order (bind, drop privileges, open the door) |
### 10.3 Crates to build or finish
| Crate | Purpose |
|---|---|
| **net-io** | transport substrate (this crate) |
| **bouncer-io** | the door: ban memory, in-flight revocation, egress wipe-and-eject; implements `Gate` |
| **tls-net** | adapter from rustls to net-io's TLS seam; we do not write TLS (D-003) |
| **serve-net** | network service module on net-io: ingress modes, sessions, drain |
| **fcgi-net** | FastCGI engine, sans-IO |
| **customs-sec**, **gate-net**, **gate-sec**, **shield-sec**, **sentinel-sec**, **sandbox-sec** | security stack (each has its own plan) |
| **quic-net**, **dns-net** | later |
### 10.4 Dependency hygiene elsewhere
- `network-protocol` 1.4.0 overlaps net-io and carries about 28 third-party dependencies; net-io supersedes it. Two lessons taken: its Unix server removes an existing socket path unconditionally before binding (net-io removes only a stale socket: FR-53, T-24), and its named-pipe module claims a throughput advantage over loopback TCP without a benchmark (net-io measures before claiming).
- `service-kernel`, `proc-daemon`, `signal-mod`, `config-lib` depend on tokio or other third-party crates; tracked in their own plans.
---
## 11. Testing and verification
| Layer | What | Where | From |
|---|---|---|---|
| Unit | every function, every `Ok` and `Err` path | bottom of each file | 0.3 |
| Socket integration | real loopback sockets for every transport, option, limit, hook, close reason | `tests/` | 0.3 |
| Hostile peers | slowloris, slow-read, half-open, never-finishing handshakes, floods at the cap, PROXY attacks, kill races | `tests/server.rs`, `tests/edge.rs` (PROXY and handshakes from 0.4) | 0.3 |
| Fault injection | `sys` shim schedules (appendix A), system calls per round trip | `tests/shim.rs` (cfg `net_io_shim`); thread-scoped rules in unit tests | 0.3 |
| Property | PROXY parser (whole, split at every byte, byte by byte, arbitrary), CIDR and prefix keys, wheel against a sorted-list model, pool against a model, completer against a model, manifest round trip | seeded generator in `src/testing.rs` (D-076) | 0.3 / 0.4 |
| Fuzz | `proxy_v1`, `proxy_v2`, `proxy_any`, `handoff_manifest`, `systemd_env`, `cidr_parse`, `trusted_proxies_parse` | cargo-fuzz, corpus outside the repository | 0.4; continuous from 0.7 |
| Concurrency | loom: door word, closing flag, completer ring, id blocks, kill vs completion generation, wake-pending flag | `cfg(loom)` | 0.3 |
| Miri | non-`sys` modules with the shim | nightly CI | 0.3 |
| Sanitizers | ASan, TSan on the integration suite | Linux nightly CI | 0.5 |
| Allocation | counting allocator on every hot path after warm-up | `tests/alloc.rs` | 0.3 |
| Leak | 100,000 connections through every close path; descriptor and handle counts back to baseline | `tests/leak.rs` (`NET_IO_LEAK_CONNECTIONS`) | 0.3 |
| Layout | every `sys` constant, size, offset, and signature against `libc` / `windows-sys` | `#[cfg(test)]` beside the declarations in `src/sys/` | 0.3 |
| Conformance | PROXY specification examples; RFC 3720 CRC32C vectors; RFC 8305 timing with `ManualClock` | `tests/conformance_*.rs` | 0.4 / 0.5 |
| Cross-process | hand-off between two real processes; systemd emulation; FastCGI emulation | `tests/process_*.rs` | 0.4 |
| Soak | 24 h mixed normal and hostile traffic (0.7), 72 h (0.9) | release candidate box | 0.7, 0.9 |
| Benchmarks | section 9 | `benches/`, `compare/` | 0.6 |
Test names: `test_<subject>_<condition>_<expected>`. Every `CloseReason` variant has one integration test that produces it on a real socket and asserts exactly one `Closed` event with that reason, plus a property test that random interleavings of kill, FIN, RST, timeout, and drain always produce exactly one closed event.
---
## 12. Open decisions
James's calls. Each can be answered in one line. Technical decisions are in the log (section 14).
| Id | Question | Recommendation |
|---|---|---|
| O-1 | REPS says "MUST use tokio". net-io records an exception (D-047). Should the REPS rule itself change portfolio-wide to "no rented runtime"? | yes; the portfolio standard already calls the clause a known mistake, and every network crate follows the no-runtime rule |
| O-2 | Hosting: does the cPanel server have WHM/root, or is it shared hosting? | if shared, FastCGI moves into the first run of the web stack |
Closed at 0.6: OD-26 (FastCGI on Windows) moved after 1.0, as no Windows FastCGI deployment is planned (D-139).
Closed at the design lock: O-3 (publish to claim the name) is settled by the release routine, which publishes 0.2.0 once James approves the API (D-048). O-5 (users core name) belongs to the gate crates' plans and moved there.
---
## 13. Ideas parking lot
Ideas worth keeping that are not scheduled. Each moves into the roadmap with a reason, or is dropped with a reason.
- **Kernel-level ban enforcement (Linux):** the gate pushes banned prefixes into an eBPF/XDP filter so banned traffic is dropped before it costs a system call. net-io's 1.0 idea (`SO_ATTACH_REUSEPORT_CBPF`) is the first step.
- **kTLS offload (Linux):** after the handshake, the kernel does record encryption (P3).
- **TCP Fast Open client side, `TCP_DEFER_ACCEPT`, FreeBSD accept filters:** fewer round trips; no worker woken until data arrives (P2/P3 in the option table).
- **Reverse-proxy client mode (`proxy-net`):** our own server forwarding to backends, using `encode_v1`/`encode_v2`.
- **HTTP/2 and HTTP/3 engines** on top of net-io and quic-net.
- **Connection fingerprint library:** ClientHello (JA4-style) and TCP-level fingerprints shared by bouncer-io and sentinel-sec.
- **Faster refused connects on Windows:** `SIO_TCP_INITIAL_RTO` with no SYN retransmissions reports a refused connect in under 0.5 ms instead of about 2 s (spike S-5), at the cost of retry behaviour on lossy networks; a candidate `ConnectOptions` setting.
- **Hardware CRC32C** (SSE4.2 `crc32`, ARMv8 CRC): only if the PROXY benchmark shows CRC is a measurable share (it is unlikely to be at 1 KiB headers).
- **A public timing wheel** (`Wheel`), if another crate needs one.
---
## 14. Decision log
| Id | Date | Decision | Reason |
|---|---|---|---|
| D-001 | 2026-10-13 | One crate for all transports (TCP, UDP, local), not `tcp-net` / `udp-net` | the driver, addresses, limits, timeouts, and drain are shared; splitting duplicates them or forces a third crate |
| D-002 | 2026-10-13 | QUIC goes to `quic-net`, DNS to `dns-net`, both later | QUIC builds TLS 1.3 into the transport (a large separate job); `std` covers DNS for now |
| D-003 | 2026-10-13 | TLS through a seam; rustls behind a separate `tls-net` adapter | keeps net-io dependency-free; TLS is the accepted third-party exception |
| D-004 | 2026-10-13 | No tokio, no async runtime; blocking pool first, own reactor second | ours, not rented; the sans-IO engines above work with either driver |
| D-005 | 2026-10-13 | Internal driver interface is completion-shaped | IOCP and io_uring are completion-based; readiness backends adapt cheaply, the reverse does not |
| D-006 | 2026-10-13 | net-io defines gate, TLS, and event traits; implementers depend on net-io, never the reverse | one-way dependencies; protocol-free rule |
| D-007 | 2026-10-13 | Mandatory gate check before every socket write, with wipe-and-replace and abortive close | no byte leaves a revoked connection even if a handler skips its checkpoints |
| D-008 | 2026-10-13 | Template aligned with http-net: edition 2024, MSRV 1.85, `Apache-2.0 OR MIT`, `dev/` committed | portfolio consistency |
| D-009 | 2026-10-13 | Naming: `gate-net` for authentication and sessions, `gate-sec` for authorization, `bouncer-io` for the door, `customs-sec` for contract inspection | James's naming |
| D-010 | 2026-10-13 | `shield-sec` and `sentinel-sec` (`sentinel` is taken) | public crate names in the `-sec` family |
| D-011 | 2026-10-13 | "Protocol parser" in the request path is a role, not a crate | each protocol keeps its own engine above the protocol-free layers |
| D-012 | 2026-10-13 | We never write our own TLS; `tls-net` stays a thin adapter | TLS safety comes from years of audits and side-channel work |
| D-013 | 2026-10-23 | The library specification written for the 0.2 design lock is adopted as the design contract; this plan, `ROADMAP.md`, and `docs/API.md` were rewritten from it | one contract for every later version; the specification extended the plan (drivers, sessions, completer, control line, hand-off, inheritance, limits defaults) and differed in a few places, each resolved below |
| D-014 | 2026-10-23 | OS access through our own `extern` declarations in `src/sys/` for x86_64 and aarch64 on linux-gnu, linux-musl, apple-darwin, unknown-freebsd, pc-windows-msvc, pc-windows-gnu; `libc` and `windows-sys` only as dev-dependencies for a CI layout cross-check (constants, sizes, offsets, signatures); other targets build the blocking driver over `std` and return `Unsupported` (was open decision OD-2) | zero runtime dependencies is the portfolio's rule; the surface is about 80 functions and 150 constants [est.], small enough to own (frozen in `SYS-SURFACE.md`); the cross-check removes the main risk, ABI drift; the cost is hand-maintained declarations, paid once per target |
| D-015 | 2026-10-23 | io_uring after 1.0 as P3 behind the non-default feature `uring`, with automatic fallback to epoll; never the default (was OD-4) | measured gains are workload-dependent (about +10% at 1,000 connections with mitigations on, a loss without them, per the Alibaba/OpenAnolis analysis) and its security record (kCTF VRP, Docker's seccomp profile, `io_uring_disabled`) argues against a default; the blocking pool and the reactor carry 0.3 to 1.0 |
| D-016 | 2026-10-23 | On Windows a listening socket is never associated with an I/O completion port, in any driver or process. The reactor accepts on Windows with one accept thread per listener (`WSAPoll` on the listener and a wake socket, then non-blocking `accept`); accepted sockets join the workers' IOCP ports. `AcceptEx` is not used for listeners; `NtSetInformationFile(FileReplaceCompletionInformation)` is rejected (was OD-5) | spike S-1: the association belongs to the shared kernel socket, so once any process associates a listener, a process that receives it by hand-off can never associate it (error 87, even after the first process closes), and its `AcceptEx` completions are delivered to the first process's port with a pointer from the wrong address space; a listener never associated works normally in the receiver. The native-API workaround redirects the old process's in-flight I/O. Accept rate is not the bottleneck on Windows deployments [est.]; at 0.6 `AcceptEx` may be offered for listeners that opt out of hand-off if a measurement shows it is worth a second path |
| D-017 | 2026-10-23 | Full-word names before anything is published: `TcpConnection`, `UnixConnection`, `PipeConnection`, `LocalConnection`, `ConnectionInfo`, `TcpOptions`, `SessionContext`, `Input`, `Output`, `Buffer`, `BufferPool`, `OutputFull`, `Endpoint`, `PeerCredentials`, `OsErrorClass`, `Operation`, `OperationKind`, `Rejected`, `Descriptors`, `DatagramBuffer`, `OutgoingDatagram`, `ProxySsl`, `ProxyAddress`, `ProxyProgress`; `SizeClass` constants `SMALL`, `MEDIUM`, `LARGE` (was OD-31; full table in `docs/API.md`) | the standard requires plain words without abbreviations; renaming is free before the first publish and expensive after; established socket vocabulary (`nodelay`, `backlog`, `linger`, `keepalive`, `Tlv`, `Cidr`) stays because it is the plain word for the thing |
| D-018 | 2026-10-23 | The gate seam is one trait, `Gate`, with an associated `Ticket` returned by `admit`, passed by `&mut` to `readmit` and `check`, and returned by value to `closed(ticket, info, reason)`; `check` returns `Result<(), Denial { code }>` and the connection's `Session::on_denied(code, output)` formats the refusal; `AllowAll` is the explicit no-gate. Agreed with bouncer-io for both 0.2 locks (its O-6, option (a)), with the additions in D-051 | the ticket removes a lookup by id on every write (bouncer measured the alternative at 5 to 10 ns per write [est.]); one trait means one builder call and no `Egress<Token = ...>` bound; `closed` gives the gate releases and strikes without parsing events; the session, not the gate, knows the protocol's framing state, which keeps the gate protocol-free |
| D-019 | 2026-10-23 | TLS seam in place, after tls-net's specification: `TlsAcceptor` (per-connection session) and `TlsSession::step(input, output)` with `TlsEvent::{NeedInput, Transmit, HandshakeComplete, Data, OutputFull, PeerClosed, Closed}`, `encrypt`, `close`, `on_transport_closed`, `info`, `input_capacity_required`. ClientHello inspection belongs to the engine; net-io drops its own hello capture and hook and maps `TlsFailure::HelloRefused` to `CloseReason::HelloRefused` | cross-crate decision X-2 (James, 2026-10-21), conditional on rustls reassembling multi-record handshake messages inside the caller's buffer: confirmed in rustls 0.23.46's source (section 6.15); no copy between engine and connection; the engine already parses the hello, so a second capture and hook would cost a copy and a second parse |
| D-020 | 2026-10-23 | `Server` is not generic; the factory, gate, sink, and acceptor types live in the worker loops | the type a program holds stays `Server` (simple API); nothing outside the workers needs those types |
| D-021 | 2026-10-23 | `Server::wait_for_stop` and `ServerHandle::request_stop` added | a Tier 1 program needs a one-line way to run until stopped and then drain; without it every user writes a parking loop |
| D-022 | 2026-10-23 | Any `Fn(&ConnectionInfo) -> S` is a `SessionFactory` (blanket impl) | `serve_sessions(addr, \|_\| Echo)` instead of a factory type for the common case |
| D-023 | 2026-10-23 | `Step::Close` always flushes, then FIN and the bounded linger; `Step::Abort` resets; `CloseAfterFlush` removed. Default `on_wake` returns `Abort`; default `on_timeout` returns `Close`, and a close from `on_timeout` carries the timeout's own reason and close style (amended by D-059) | one obvious way to close gracefully; an unexpected wake fails closed; a timeout keeps its own reason so logs and the gate see `Idle` or `ReadTimeout`, not a session error |
| D-024 | 2026-10-23 | Unix sockets default to mode `0600`; naming a group makes it `0660` (was OD-25, where the plan said `0660`) | fail closed: with no group named, `0660` would grant the process's primary group, which nobody chose; one call (`LocalOptions::group`) gives group access when it is wanted |
| D-025 | 2026-10-23 | A per-source concurrency cap lives in net-io (keyed by `PrefixKey`, table sized so it cannot overflow); rates and bans stay in the gate (was OD-8) | needs no policy, costs one table update per accept, and keeps a server safe without a gate |
| D-026 | 2026-10-23 | `ipv6_only` on by default (was OD-9) | an IPv6 listener is IPv6 only; dual-stack is explicit; no IPv4-mapped surprises in caps and bans |
| D-027 | 2026-10-23 | Wipe buffers on release by default; `wipe_buffers(false)` to turn off (was OD-10) | peer bytes may carry credentials; 50-100 ns per 2 KiB [est.] |
| D-028 | 2026-10-23 | Blocking-driver defaults: 128 workers, 256 KiB stacks, 5 s idle, `pressure()` = busy workers (was OD-11) | an idle connection holds a worker; revisit with the first consumer's measurements |
| D-029 | 2026-10-23 | The `Driver` trait becomes public Tier 3 at 0.5, marked unstable until 1.0, frozen at 1.0 (was OD-12) | an executor adapter and future drivers need it |
| D-030 | 2026-10-23 | The executor adapter is an optional, default-off feature `executor` in net-io, depending on the executor crate by published version (was OD-13) | one place for the adapter; never in the default build |
| D-031 | 2026-10-23 | MSRV stays 1.85; OS features newer than `std` 1.85 come through `sys` (was OD-14) | portfolio MSRV |
| D-032 | 2026-10-23 | Close style: RST for kill, gate denial, refusal, drain deadline, slow peer, limits, timeouts, PROXY and TLS failures, session errors; graceful for peer close, idle, drained, session close. A denial after part of a response left is a reset with no denial bytes (was OD-15, OD-16) | a stream cannot be repaired mid-response; RST frees kernel resources at once where the peer is misbehaving |
| D-033 | 2026-10-23 | `EventSink` is a type parameter of the builder, default `NoopSink` (was OD-17) | zero cost unused, one static call used |
| D-034 | 2026-10-23 | UDP amplification factor 1 by default, configurable up to 3 (was OD-18) | strict by default; QUIC sets 3 for its handshake per RFC 9000 |
| D-035 | 2026-10-23 | FreeBSD CI through a VM action on GitHub runners from 0.5; the action is chosen then (was OD-19) | no self-hosted runner to maintain |
| D-036 | 2026-10-23 | `LISTEN_*` variables are not unset after reading (was OD-20) | `remove_var` is `unsafe` in edition 2024 because it races other threads; the pid check already stops a child from using them |
| D-037 | 2026-10-23 | Minimum platforms: Linux 4.18 with run-time detection above it, macOS 11 (no hard CPU affinity: `Unsupported`), Windows 10 1809 / Server 2019, FreeBSD 13; io_uring (P3) needs Linux 6.1 [to confirm when built] (was OD-21, OD-22, OD-23) | covers RHEL 8, every supported macOS, and every Windows still in support; older systems get `Unsupported`, never undefined behaviour |
| D-038 | 2026-10-23 | net-io has its own minimal `Clock` trait (was OD-24) | no first-party dependency for one method; an application adapts `clock-lib` in one line |
| D-039 | 2026-10-23 | No `no_std` core crate now; the OS-free modules (PROXY, CIDR, prefix keys, CRC32C, wheel) stay `core`-clean, checked in CI (was OD-27) | split only when a second consumer needs them without the OS parts |
| D-040 | 2026-10-23 | PROXY v2 address-and-TLV cap 1,024 bytes by default, configurable up to 65,519 (was OD-28; the plan said 4 KiB) | covers a UNIX address block, a 128-byte unique id, ALPN, authority, and SSL sub-TLVs with room to spare; smaller is cheaper to bound |
| D-041 | 2026-10-23 | `ServerBuilder::build` requires a gate; `AllowAll` must be named; Tier 1 `serve` and `serve_sessions` use `AllowAll` and say so (was OD-29) | "no gate" is a visible choice in code, never a silent default |
| D-042 | 2026-10-23 | During a hand-off both processes may accept for a short overlap (was OD-30) | no refused connection during a restart; each connection still belongs to one process |
| D-043 | 2026-10-23 | `listen_local` and `connect_local` use a named pipe on Windows; AF_UNIX on Windows stays available through `UnixListener` (was OD-7) | native, ACLs, the client pid, every supported Windows |
| D-044 | 2026-10-23 | Happy Eyeballs at 0.5 (P2), simplified for `std` resolution, behind `ConnectOptions::happy_eyeballs(true)`, default off; the Resolution Delay arrives with `dns-net` (was OD-6) | `std` returns both families at once, so the rest of RFC 8305 applies as written |
| D-045 | 2026-10-23 | The fault-injection shim is compiled under `cfg(test)` or the custom cfg `net_io_shim`, never as a Cargo feature | a cfg cannot appear in the published feature list or be enabled by a dependent; normal builds contain no dispatch |
| D-046 | 2026-10-23 | Windows randomness for the per-source hash key through `BCryptGenRandom` with `BCRYPT_USE_SYSTEM_PREFERRED_RNG`; Linux and FreeBSD `getrandom`, macOS `getentropy` | documented interfaces only (`ProcessPrng` is what `std` uses but is not documented for direct use). Not needed by the blocking driver: D-062 |
| D-047 | 2026-10-23 | net-io records the REPS "MUST use tokio" clause as an approved exception (portfolio standard 2.8) | no async runtime in this crate (D-004); the portfolio-wide REPS change stays James's open decision O-1 |
| D-048 | 2026-10-23 | The crate name is claimed by publishing 0.2.0 in the release routine after James approves the API; 0.1.0 is not published separately (closes O-3) | the release routine publishes every completed version; a separate scaffold publish adds nothing |
| D-049 | 2026-10-23 | `TcpConnection::set_gate` registers dialed connections with a gate (`AdmitStage::Dialed`), replacing the earlier `set_egress` | dialed back-channel connections must be revocable like accepted ones, which needs the same ticket and the same close notification |
| D-050 | 2026-10-23 | `Refusal::WithBytes` bytes are `&'static`, at most 512, written once without retry and only on TCP after the handshake | canned replies need no allocation; the bound and the single attempt keep the refusal path from becoming an amplifier or a blocking point |
| D-051 | 2026-10-23 | Keep an opaque `GateTag(u64)`: returned in `Admitted::tag`, stored in `ConnectionInfo::gate`; and a per-response tag the session sets with `Output::begin_tagged_response(tag)`, seen by the gate as `WriteCheck::response` and cleared by `end_response` (requested by bouncer-io) | a protocol engine must name the connection to the gate (to open request tickets) without being generic over the gate type; on HTTP/2, or a proxy pooling many clients on one connection, only the request's own tag lets a write check judge the right request; cost: 8 bytes per connection and one field load per check |
| D-052 | 2026-10-23 | Blocking-driver waits per platform. Unix: blocking `recv` with `SO_RCVTIMEO`; sends with `MSG_DONTWAIT`, then `poll` on the socket and the worker's wake pipe on `EAGAIN`. Windows: `SO_RCVTIMEO`/`SO_SNDTIMEO` are never used; sockets are non-blocking and every wait is `WSAPoll` on the socket and the waiter's wake socket. Wake channels: one per worker (a pipe on Unix, a loopback UDP socket connected to itself on Windows), and for connections used outside a server one per calling thread, created on first use | keeps one system call per read on Unix; on Windows the documented "indeterminate state" after a timeout (spike S-2 saw no loss, but a lost byte mid-stream is a framing desync) is avoided at the cost of one extra system call per read; kills wake stalled sends at once on both (NFR-19); the wake channels are counted in the descriptor-limit check (FR-61) |
| D-053 | 2026-10-23 | Kill guard: the state word's `KILLING` bit keeps the owner from closing a socket or dropping a wake channel while a killer's wake call on it is in flight | a killer's `shutdown` or wake write on a descriptor the owner already closed could hit a reused descriptor number; the guard costs one compare-and-swap per kill and a bounded wait on close only when a kill is in flight |
| D-054 | 2026-10-23 | Windows hand-off is made single use by net-io: a 128-bit random nonce per manifest, refused on reuse in the receiving process; `SharedListeners::write_to(self, ..)` consumes the value | spike S-1: Windows returns the same socket for a reused `WSAPROTOCOL_INFOW` and a dead one for a blob used after the sender closed |
| D-055 | 2026-10-23 | Per-connection options: not re-applied on Linux and on Windows (every tested option is inherited from the listener with plain `accept`, spike S-4); applied explicitly to each accepted socket on macOS and FreeBSD until their CI tests show inheritance; a CI test asserts inheritance on Linux and Windows so a platform change is caught | fewer system calls per accept where the OS already does the work; no unverified assumption elsewhere |
| D-056 | 2026-10-23 | Windows connect completion through `WSAPoll`; `select` is dropped from the OS surface | spike S-5: `WSAPoll` reported every refused connect (`POLLWRNORM \| POLLERR \| POLLHUP`, `SO_ERROR` 10061) on Windows 10.0.26200; the older reports of missed refusals did not reproduce; the CI runners re-check it in the 0.3 tests and `select` returns only if one of them misses. Windows reports a refused loopback connect after about 2 s (SYN retries); shortening it (`SIO_TCP_INITIAL_RTO`) changes retry behaviour on lossy networks and goes to the parking lot |
| D-057 | 2026-10-23 | net-io never calls `shutdown` on a listening socket; accept threads are woken only through their wake channel | spike S-3: on Linux, `shutdown(SHUT_RD)` on a listener wakes `accept` with `EINVAL` and disables the listener for good (new connects are refused), which would break hand-off |
| D-058 | 2026-10-23 | `UnixListener::bind_atomic` (bind under a temporary name, then `rename` into place) confirmed on Linux; the listener records its final path itself | spike S-6: `rename` was atomic on ext4 and tmpfs (about 14 million connects racing 4,500 renames over a stale socket file never saw `ENOENT`), the old name gives `ENOENT` after the rename, and `getsockname` still reports the temporary name; macOS and FreeBSD are checked by the 0.4 CI test |
| D-059 | 2026-10-23 | Changes from an independent review of `docs/API.md` before approval. Names: `Limits` timeouts end in `_timeout`; `Server::wait_for_stop` and `ServerHandle::request_stop`; `Session::on_denied`; `TlsSession::on_transport_closed`; `ConnectionInfo::socket_peer`, `Endpoint::socket_addr`, `NetError::os_error_code`; `UnixListener::bind_atomic`; `LocalOptions::group_access`; `ControlState::Shed { from_lane }`; `TcpOptions` (was `ConnectionOptions`, too close to `ConnectOptions`); `receive_buffer`. Kept on purpose: `recv`/`send`, `addr`, the door. Behaviour: `Limits` records only what was set and the driver's defaults fill the rest (`resolve`), with dependent defaults following `max_connections`; a builder without a gate or a workload has no `build` (compile error naming the missing part, sealed `Workload`); `Server` is `#[must_use]`; `release_listeners` works with the door open so hand-off can overlap (D-042); `NetError` gains `From<io::Error>`, `From<AddrError>`, `FileTruncated`, `Address`, and loses `attempts`; every error type implements `Error`; `Output::push_partial`, and after `Step::Read` net-io calls `on_data` again when the input changed; dropping a connection never blocks; `begin_tagged_response` replaces `begin_response(Some(..))`; constructors for implementers and tests (`TlsError::new`, `TlsInfo::new`, `ConnectionInfo::for_testing`, `ConnectionId::new`, `WriteCheck::new`); `encode_v2` takes parts instead of a parsed header; data types that always-built items refer to are compiled in every feature set; `UdpSession`, `SendChunks`, the batch and multicast types, and per-cause `Stats` declared; `client_addr` and `client_key` move to 0.3 | the review found examples that did not compile, a seam that could not be implemented outside the crate, an API order that contradicted D-042, defaults that changed with a Cargo feature instead of the driver, and names a reader could not decode; the simplified-API rule (standard 2.10) makes each of these a defect |
| D-060 | 2026-10-23 | `Gate::readmit` has no default implementation | a default `Ok(())` would let a gate that never thought about proxies admit every real client behind a trusted proxy on the proxy's verdict; writing one line is the visible choice |
| D-061 | 2026-10-23 | `Endpoint::Unix` and `Endpoint::Pipe` hold `Arc<UnixPath>` / `Arc<PipeName>` shared with the listener; `ConnectionInfo::proxy` and `::tls` are `Option<Arc<ProxyFacts>>` / `Option<Arc<TlsInfo>>` drawn from per-worker pools and returned when the connection closes | with the facts inline, `ConnectionInfo` measured about 1.8 KB (two endpoints sized for a 256-unit pipe name, 420 bytes of PROXY TLV copies, 290 bytes of TLS facts), which alone breaks the 512-byte idle-connection target (B-5); shared and pooled, it is near 200 bytes and still allocates nothing after warm-up |
| D-062 | 2026-10-23 | Per-source table on the blocking driver: 16 shards of `Mutex<HashMap<PrefixKey, u32>>` with std's `RandomState` (SipHash keyed from OS randomness once per process), sized from `max_connections` at build | the accept path is one thread per listener and a close is one lock per connection, so a shard lock costs tens of nanoseconds uncontended; std's keyed hash gives the "no attacker-chosen collisions" property without new FFI (`getrandom` / `BCryptGenRandom` leave the 0.3 OS surface); the open-addressing table with striped compare-and-swap counts is measured against shards on the reactor in 0.5 |
| D-063 | 2026-10-23 | The timing wheel ships with the reactor in 0.5, not in 0.3 | a blocking worker holds one connection with at most five deadlines, all computed from its progress time and carried by `SO_RCVTIMEO` or the wait timeout; a wheel pays off only where one worker holds thousands of connections. Edge cases 53 to 60 and benchmark B-8 move with it (roadmap record) |
| D-064 | 2026-10-23 | FreeBSD and the std-only fallback arrive in 0.5; 0.3 builds on Linux, macOS, and Windows (x86_64, aarch64) and names the rest in a `compile_error!` | both need the reactor's kqueue path or a second blocking path without our FFI; a clear compile error beats a silent partial build |
| D-065 | 2026-10-23 | When net-io chooses the phase (an accepted connection whose owner never called `set_phase`), a reply returns the connection to `Idle` once no input is left over; `set_phase` turns this off; dialed connections never get it | the API promises that a session which never sets a phase still gets idle and read deadlines; without the return, a keep-alive session sat on the 30 s read timeout forever and a drain could not see it as idle |
| D-066 | 2026-10-23 | A graceful close skips the linger when the connection is idle and a zero-timeout poll shows nothing unread | the linger protects unread request bytes from turning the close into a reset; an idle connection has none, and lingering made a drain of idle connections wait out `graceful_linger` (2 s). Cost: one poll per idle close |
| D-067 | 2026-10-23 | The blocking hand-over refuses with `Busy` when queued plus busy connections reach the worker count; a worker counts as busy from the moment it takes a job (under the queue lock) until its close | the first rule ("queue length at least the idle workers") counted idle workers from zero, so a fresh server refused its first connection, and a worker about to look for work counted as busy. Cost: one atomic load under the existing lock |
| D-068 | 2026-10-23 | A session that closes after the peer's FIN (the default `on_peer_closed`) closes with `PeerClosed`, not `SessionClosed` | appendix A case 1; the peer ended the connection |
| D-069 | 2026-10-23 | Per-worker counters sit behind a single-writer sequence lock; `stats()` sums consistent per-worker snapshots | appendix A case 132: a reader could see a close's `bytes_out` without its `bytes_in`. Cost: two stores per close for the worker (its counter updates became plain load and store, cheaper than the read-modify-writes they replace); a reader retries a torn read |
| D-070 | 2026-10-23 | An accepted connection whose peer address is unspecified or has port 0 is reset at once and counted as an `AcceptError` | appendix A case 47: nothing to key caps or the gate on |
| D-071 | 2026-10-23 | The fault-injection shim gains thread-scoped rules (unit tests run in parallel threads), the `Reserve` and `RecvTimeout` call sites, and per-site call counters; integration tests reach it through a `#[doc(hidden)] pub mod __shim` that exists only under `cfg(net_io_shim)` | a global rule in one unit test could be consumed by another test's socket call; the counters measure system calls per round trip (B-13) without strace; the hidden module is never in a normal build and is not part of the API |
| D-072 | 2026-10-23 | The minimum-rate check requires `bytes_per_sec` over the time its buckets actually cover (the full older buckets plus the elapsed part of the current one, never more than the time since activation), not over the whole window | found by the property test: with a window of few buckets, right after a bucket boundary the check demanded a full window of bytes from a window of history and flagged a peer sending at twice the rate |
| D-073 | 2026-10-23 | Unix reads, including the wait for the next exchange, are one blocking `recv` under `SO_RCVTIMEO`; a drain closes idle connections through the kill slot: the owner marks the idle wait, the drain shuts the read side under the D-053 guard and wakes it, and the owner closes with `Drained`. The drain repeats this pass every 10 ms for connections that finish an exchange during it | B-13 asks for 2 system calls per round trip on Unix; polling before each idle read made it 3. Measured after the change: 2.00 on Linux, 3.00 on Windows (D-052), counted at net-io's call sites. A reply in progress still goes out (writes are untouched). Cost: two atomic read-modify-writes per idle read |
| D-074 | 2026-10-23 | The gate is asked once per write call, before its first system call, not again after `WouldBlock`; the zero-byte-send rule (appendix A case 19) covers the vectored path too | a retry sends the bytes the gate already approved, and a gate that counts bytes would count them twice; every new write call (each flush step) is still checked first, so cases 25 to 27 hold. The vectored path used to wait forever on a send that moved 0 bytes |
| D-075 | 2026-10-23 | The control line's reactions run in the accept thread: `Draining` closes the door and drains with the line's deadline (or the drain timeout); `Stopping` closes the door and kills every connection with `Shutdown`; `Shed { from_lane }` closes idle connections in shed lanes at once (`CloseReason::Shed`) and sessions in those lanes close at their next idle point | before, the reactions took effect only if the program was blocked in `wait_for_stop`, and `CloseReason::Shed` was never produced, against the API table |
| D-076 | 2026-10-23 | Property tests use an in-crate seeded generator (xorshift64*) instead of proptest | no new dev-dependency, and every failure reproduces from its seed; the properties (CIDR and prefix keys against integer masks, the output stream against a model, the rate check against fast and silent peers) need generation, not shrinking. Revisit for the PROXY parser in 0.4, where shrinking helps |
| D-077 | 2026-10-23 | `Endpoint` implements `Display` (`127.0.0.1:80`, the Unix path, the pipe name, or `unnamed`) | printing a listener's address was not possible without matching on the variant |
| D-078 | 2026-10-23 | Windows: an accepted socket gets `SetHandleInformation(.., HANDLE_FLAG_INHERIT, 0)` | std's `accept` does not clear inheritance on Windows, so a child process started by the application would inherit every client socket |
| D-079 | 2026-10-23 | Benchmark baselines in 0.3 use a plain harness (`harness = false`), medians over rotated rounds; criterion arrives with the 0.6 microbenchmarks | a baseline needs round trips per second and percentiles, not statistics machinery; rotating the order over 7 rounds and taking medians keeps scheduler placement on a multi-die CPU from moving a single number by 2x |
| D-080 | 2026-10-23 | A receive error of kind `TimedOut` closes the connection with `Io(TimedOut)`; only `WouldBlock` means "try again" | an expired `SO_RCVTIMEO` reads as `WouldBlock` on Unix and Windows sockets are non-blocking, so `TimedOut` can only be the kernel giving up on the connection (keepalive probes, appendix A case 8); retrying it looped until net-io's own deadline and reported the wrong reason |
| D-081 | 2026-10-23 | Requests from serve-net: `TrustedProxies` moves from 0.4 to 0.3, and `SessionContext::set_idle_timeout(Option<Duration>)` is added in 0.3 (this connection only, from its next idle wait, at least one tick; `None` restores the limit). Scheduled for 0.4: `Server::add_listener` / `remove_listener` (door closed until opened; removal closes that listener's door and drains its connections), trusted proxies per listener for PROXY, and the raw monotonic clock in microseconds for systemd `MONOTONIC_USEC` | serve-net 0.3's `http_behind_proxy` takes a `TrustedProxies` and the stack has one source of truth for trusted proxies; `TrustedProxies` is pure address code with no OS dependence. serve-net shortens keep-alive under pool pressure (its FR-58). The other three are needed by serve-net 0.4 (reload, systemd) and belong with the PROXY and systemd work |
| D-082 | 2026-10-26 | macOS listener check (`TcpListener::from_std`): when `SO_ACCEPTCONN` answers `ENOPROTOOPT`, a socket counts as listening if it has no peer (`getpeername` gives `ENOTCONN`) and is bound to a port | macOS does not answer `SO_ACCEPTCONN` (CI on macOS 15); `SO_LISTENQLIMIT` is not in `libc`, so its value cannot be cross-checked (SYS-SURFACE rule 4). A bound socket that never listened passes the check and fails at its first `accept` with `EINVAL`, a listener failure |
| D-083 | 2026-10-26 | macOS uses the Windows IO model in the blocking driver: sockets stay non-blocking and every receive waits in `poll` first; Linux keeps blocking receives under `SO_RCVTIMEO` | CI on macOS 15: a `sendmsg` with `MSG_DONTWAIT` on a blocking socket blocked on a full send buffer (a flush to a peer that never reads hung with no write timeout), and `setsockopt` fails with `EINVAL` once the connection has ended, which broke the per-read `SO_RCVTIMEO` update after a peer's FIN. Cost: one `poll` per receive on macOS, 3 system calls per keep-alive round trip there (B-13: 2 on Linux, 3 on macOS and Windows) |
| D-084 | 2026-10-26 | A connection the peer resets between `accept` and its setup is that connection's failure: on macOS a failed post-accept setup (`SO_NOSIGPIPE`, non-blocking mode) is reported as `ECONNABORTED` (counted, accepting continues), and a failed per-connection option closes it through the normal close path with `Io` and one `Closed` event | macOS answers `setsockopt` on a reset connection with `EINVAL`, which the accept classes read as a broken listener (the accept thread stopped), and the option path released the slot without a close event, so an admitted connection never closed (found by the exactly-once property test on macOS CI) |
| D-085 | 2026-10-26 | The per-source cap is counted per listener: each listener has its own table sized from its own limits and applies its own `max_per_source` | a listener's `max_per_source` was accepted and ignored (one table from the server limits), which serve-net needs per listener (a proxy-facing and a direct listener on one server); a host can hold the cap on each listener, which is what separate limits mean |
| D-086 | 2026-10-26 | The loom models build under the crate's own cfg, `net_io_loom`, not `loom` | a downstream crate that runs its own loom models with `--cfg loom` could not build net-io, whose loom code needs a dependency only net-io's tests have |
| D-087 | 2026-10-26 | `Stats::closes` and `Stats::refusals` iterate the non-zero counters; `WorkerId::for_testing(index)`; `Step::Read` says that queued output is not sent | feedback from the first consumer (serve-net's hello demo): reading counters meant building every `CloseReason` by hand, tests of per-worker state could only name worker 0, and an unflushed keep-alive reply cost an hour |
| D-088 | 2026-10-26 | PROXY on the server: the worker checks trust before its first read (`ProxyUntrusted`), reads the header into a pool buffer under the handshake deadline (`HandshakeTimeout`, `ProxyMissing` for non-PROXY bytes, `ProxyInvalid` for a bad or oversized header), moves the per-source slot to the real client (`Refused(SourceCap)` when that client is at its cap), sets `real_peer` and `proxy`, runs `Gate::readmit` (a refusal writes its bytes once and closes with `Refused(Gate(code))`), then emits `ProxyResolved`. Bytes after the header stay in the handshake buffer, which the connection reads before the socket and the worker takes back at close; `ProxyFacts` are recycled per worker when nothing else holds them | one code path for handlers and sessions, no copy of the bytes after the header beyond the one into the reader's buffer, and no allocation per proxied connection after warm-up (D-061); the trust check in the worker costs a worker slot for an untrusted peer but keeps the accept thread free of per-listener PROXY state. A listener's own `trusted_proxies` replaces the server's (D-081); `Required` with an empty or missing list fails at build (appendix A case 122) |
| D-089 | 2026-10-26 | The TLS seam runs inside the connection's link: with TLS on, reads return plaintext (the engine decrypts in place in a ciphertext buffer of the largest read class; net-io copies each record's plaintext once into the reader's buffer) and writes encrypt into a 64 KiB scratch buffer that is sent within the same call. The gate is asked once per plaintext write, for exactly the bytes that call encrypts (at most one record for `write`, whole records that fit the scratch buffer for a vectored flush), not again before the ciphertext flush. `close_notify` goes out on a graceful close, never on an abortive one; `peek` on a TLS connection is `Unsupported` | one TLS path for blocking handlers and sessions with the same single copy per record the API names (into the reader's buffer instead of a `copy_within` in the same buffer); a second gate check before the flush would make a byte-counting gate count every byte twice, and the ciphertext of a write goes out before the call returns, so revocation is still seen before the next write. Cost per TLS connection: the two pool buffers (64 KiB each by default) while it is open |
| D-090 | 2026-10-26 | The gate check covers only bytes it has not approved yet: after a partial send the unsent tail, which is at the head of the next offer (output drains in order), is carried as approved | D-074 stopped re-checking after `WouldBlock`, but a partial vectored send still offered its tail again, so a byte-counting gate counted it twice. Cost: one integer per connection |
| D-091 | 2026-10-26 | A Unix listener is created as socket, `bind`, `lstat` (record device and inode), `chmod`, `lchown` when asked, `lstat` again (same file), then `listen`; `bind_atomic` binds under `.<name>.<pid>.<n>` and renames into place after `listen`. An existing file is removed only when `lstat` says socket and a probe connect is refused; a live socket, a regular file, a directory, or a symbolic link fails with `StaleSocketInUse` and is left alone. `Drop` unlinks only while `lstat` shows the recorded device and inode | connects to a bound socket that does not listen yet are refused, so ordering `chmod` before `listen` closes the umask window without touching the process-wide umask (not thread-safe) and without Linux-only `fchmod` tricks; `lchown` and the identity check keep a symbolic link swapped in by someone with write access to the directory from redirecting the change, and the unlink rule keeps a restart that already rebound the path from losing its socket. Cost [est.]: two to three `lstat`, one `chmod`, and one probe connect when a file exists, once per bind; nothing per connection |
| D-092 | 2026-10-26 | `UnixConnection::connect` maps both "no such file" and "connection refused" to `NotFound`; a full backlog waits up to 10 s through `SO_SNDTIMEO` (Linux blocks there) and then fails `TimedOut { kind: Connect }` | to a client a socket file nobody listens on and a missing one are the same condition (start the server), and `ConnectionRefused` carries an IP address a Unix path does not have; an unbounded wait on a full backlog would hang a client forever |
| D-093 | 2026-10-26 | Unix connections run on the same connection core as TCP: `sys::Stream` is an enum (`Tcp`, `Unix`), `UnixConnection` wraps the core, and the server hands Unix connections to a `BlockingHandler` as a `TcpConnection` whose transport is `Unix`. On a server Unix listener the per-source cap does not apply, a `require_uid` mismatch is refused before admission with the new `RefusalCause::Credentials`, and PROXY needs no trusted list: every peer that can connect is trusted | one IO path means limits, deadlines, the kill handle, the gate, TLS, and close semantics are identical on both transports and tested once; a separate handler type for Unix would split every server API in 0.4 (a connection type shared by both is a candidate for the reactor's API, 0.5). A Unix peer has no address to count, and on a local socket the file's mode and owner are the access control a trusted list would otherwise give. Cost: one enum discriminant check per IO call, under 1 ns [est.] |
| D-094 | 2026-10-26 | Descriptor passing: `send_descriptors` attaches up to 64 descriptors to the first byte of a body sent in full (one gate check for the body); `recv_descriptors` uses the connection's normal read path with `recvmsg` in place of `recv` (on Linux blocking under `SO_RCVTIMEO` like every receive), receives close-on-exec (`MSG_CMSG_CLOEXEC` on Linux, `FD_CLOEXEC` right after on macOS), and on `MSG_CTRUNC` closes every descriptor that arrived and fails the connection with `Handoff(Truncated)` | deadlines, kills, and drains apply to a descriptor receive exactly as to any read; a truncated control message means bytes and descriptors no longer line up, so continuing would let a peer desynchronise a hand-off; macOS has no atomic close-on-exec flag for received descriptors (documented window). Cost: a 264-byte control buffer on the stack per call |
| D-095 | 2026-10-26 | `LocalOptions` is vocabulary and is built in every configuration (in `opts`); `ListenerSpec::unix` exists everywhere and fails `build` with `Unsupported` without the `local` feature or on Windows before `0.5.0`; a NUL byte in a socket path is `InvalidConfig { field: "path", reason: Malformed }` (new reason) | the rule in `docs/API.md` that features gate machinery, not vocabulary, keeps the builder's shape identical across feature sets; a NUL would silently truncate the path the kernel sees |
| D-096 | 2026-10-26 | UDP guard: sources are keyed by normalized IP address (not address and port); the send allowance is `factor x (received - sent)` over the current and previous window (default 10 s, `amplification_window`, at least 1 s); budgets are integer token buckets (micro-tokens; a datagram bucket of depth `burst`, a byte bucket of one second); the table is a hash index over a slab with CLOCK eviction of unverified entries (at most 64 looks per insertion, verification may look at all); a source that finds no room shares one overflow budget and gets no send allowance; the global budget is charged only after the source budget passes; datagrams over budget are read and dropped inside `recv_from`, which checks its deadline after each drop (a flood returns within twice the read timeout); ICMP reports (`ECONNREFUSED`, `ECONNRESET`) are counted and skipped, and Windows has `SIO_UDP_CONNRESET` off | a spoofer varies ports, and the address is what gets flooded, so keying by port would hand out a fresh allowance per port; two windows avoid refusing the reply to a request that arrived just before a boundary; integer buckets cost no floating point and cannot drift; strangers that overflow a full table could otherwise pool allowance and aim it at a victim, so they get none (the cost lands only when 65,536 recent or verified sources hold the table). Cost [est.]: one SipHash lookup and two bucket updates per datagram (under 30 ns), up to 8.5 MiB for a full default table (about 130 bytes per source) |
| D-097 | 2026-10-26 | Listeners are a registry: each has an `Arc` of its settings and counters (limits, socket options, PROXY, TLS, its connection count, its per-source table, its own door, a removed flag, its wake channel) that its accept thread and every connection it admitted hold; the shared list is a mutex taken only on cold paths. `Server::add_listener` and `remove_listener` take `&mut self` (so `listeners()` still returns a slice); an added listener's door stays closed until the next `open_door`, which opens every listener's door; ids are never reused; removal closes the socket first, then drains that listener's connections with the server's drain rules (idle now, the rest killed at the deadline), and connections still queued for it close unread with `Drained` | serve-net's reload (D-081) changes listeners without restarting; workers indexing per-listener vectors by position could not survive a removal, while an `Arc` per connection keeps a removed listener's limits and cap slots valid until its last connection closes, with no lock on the accept or close path. Cost: one `Arc` clone per accepted connection (under 5 ns [est.]); one wake channel and one accept thread per listener, as before |
| D-098 | 2026-10-26 | Inheritance verifies every passed descriptor (open, a socket, stream, listening, TCP or Unix) before taking any, takes each number once per process (systemd's list is cached behind a lock; `systemd` takes what is left, `systemd_named` takes one), and sets close-on-exec. A `LISTEN_PID` that is not a number is "not for us" (an empty list, as for another pid); `LISTEN_PIDFDID` is compared with the inode of `pidfd_open(getpid())`. UDP sockets and, without the `local` feature, Unix sockets are `WrongKind`. `ListenerHandle` carries kind, name, local endpoint, and the owned socket; `ListenerSpec::adopt` uses it without binding (backlog unknown); an adopted or released Unix socket's file is never removed by this process | two owners of one descriptor number would close it twice; taking after a full verification means an error leaves the process's descriptors as systemd passed them; the pid already guards against stale variables, and a malformed pid means the variables are not ours to act on. A process that adopted a path cannot prove the file at it is its socket (the socket's inode is not the file's), so cleanup is left to the next bind's stale check. Cost: a few system calls per descriptor, once |
| D-099 | 2026-10-26 | The hand-off manifest carries kind, name, and local address per listener and no options digest; the header carries the total length so the receiver reads exactly one message; descriptors may arrive only with the first byte (a later read that brings any fails with `CountMismatch`); the receiver checks credentials before reading, reads under the caller's deadline through the channel's exchange deadline, and maps a closed channel mid-message to `Truncated` | the address check against `getsockname` already catches a descriptor swapped for another listener, while socket options have no portable set both sides read identically, so a digest would add false failures, not safety; a length prefix bounds the read and makes a short message detectable. Cost: under 12 KB of manifest for 64 listeners and one CRC32C over it |
| D-100 | 2026-10-26 | `Server::adopt` admits a dialed `TcpConnection` through the same path as an accept (caps, `AdmitStage::Adopt`, the named listener's limits, `Busy`), returning `Refused { cause }` (the connection reset) instead of emitting only an event; a gate the connection had from `set_gate` is closed with `SessionClosed` first; a TLS connection or one with bytes read ahead cannot be adopted (`InvalidConfig { field: "connection" }`). `TcpConnection::set_gate` admits with `AdmitStage::Dialed`; a refusal aborts the connection (later calls return `Closed`) and `closed` is not called for a connection the gate never admitted | one admission path keeps adopted connections under the same caps and revocation as accepted ones; the bytes inside a TLS session or a read-ahead buffer would be lost on the move, so those connections are refused rather than silently corrupted |
| D-101 | 2026-10-26 | `monotonic_micros()` reads `CLOCK_MONOTONIC` (Linux, macOS) or the performance counter (Windows) directly | systemd's `MONOTONIC_USEC` must be `CLOCK_MONOTONIC`, which `Instant` does not expose as a number (serve-net 0.4, D-081) |
| D-102 | 2026-10-29 | A pre-1.0 polish milestone (0.9.x, before any 1.0 release candidate) revisits every accepted gap, deferral, known issue, platform note, and missed target, runs a fresh performance pass and a fresh security and edge-case review, and ends in a 1.0 readiness note for James. Every gap is entered in a register in `dev/ROADMAP.md` when it is accepted, and an entry closes only with its fix or a measured reason | James's standing rule for every crate ("go back and fix any issues / perf tune / etc before we get close to release"); a register filled when a gap is accepted means nothing accepted mid-series becomes final without being looked at again |
| D-103 | 2026-11-02 | The public `Driver` is the reactor's real interface: one event loop over completions on every OS, with `ReadyDriver` adapting epoll and kqueue. Readiness is remembered per direction; a send tries first; `Close` and every other operation end in one completion | one state machine per connection, not one per backend; a round trip on epoll stays one wait, one receive, one send |
| D-104 | 2026-11-02 | Accepts stay on one thread per listener on the reactor, sharing the admission path with the pool; a connection goes to the least-loaded worker through a bounded inbox of 4,096, and a full inbox refuses with `Busy`. `EPOLLEXCLUSIVE` per worker is measured at 0.6 (register R-19) | D-016 needs the thread on Windows anyway; one admission path is one place for caps and the gate. Cost: a socket crosses threads once per connection, an estimated 1 to 3 us [est.], and no per-event cost |
| D-105 | 2026-11-02 | One timer per connection, at its nearest deadline; moving it is lazy, and the deadline is computed again when it fires. At most 1,024 timers per loop turn | an exchange costs one comparison (4 to 5 ns), and a clock jump cannot starve IO |
| D-106 | 2026-11-02 | The connection table is 16 mutex shards of `id -> slot, generation, worker, index`, touched only on open, close, kill, and `Completer` lookups | a kill arrives on any thread; the data path never takes these locks |
| D-107 | 2026-11-02 | A connection's limits are an `Arc<Effective>` shared with its listener, a `Rate` holds no settings and at most 10 buckets, and TLS state is boxed. 848 bytes, against 1,384 (register R-18) | security unchanged: the same limits are checked, and the bounds hold. Cost: one contended reference count per accept [est. 10 to 40 ns], and one pointer on a read of the limits. A longer rate window widens its buckets, so its resolution is the window over 10 |
| D-108 | 2026-11-02 | TLS is a sans-IO machine in the connection core (`tls_advance`), driven by the blocking wait or by reactor completions | both drivers share one implementation of a security-critical sequence |
| D-109 | 2026-11-02 | On IOCP an operation is a heap block, starting with its `OVERLAPPED`, that owns its buffers; `Box::into_raw` gives it to the kernel and only a dequeued packet (or a call that finished at once) takes it back. Blocks are pooled per worker. `deregister` with operations out cancels them, and their packets are dropped. At `Drop`, what is still out after a second stays allocated | T-18 by construction: no path frees memory the kernel may write. Cost: one block per connection (about 150 B), no allocation after warm-up |
| D-110 | 2026-11-02 | A connection with no held bytes waits with `ReceiveZero` in any phase, not only between exchanges, and its buffer goes back to the pool | a peer that stalls mid-exchange cannot pin a buffer. Cost: nothing on epoll and kqueue beyond the check; one more zero-byte receive per consumed read on Windows (register R-22) |
| D-111 | 2026-11-02 | An empty buffer budget pauses a read: the connection joins a per-worker list, oldest first, and the worker retries every 10 ms while anything waits, because buffers free on any worker. The read deadline keeps running. PROXY and TLS setup still close with `Limit(BufferBudget)` | case 11: a burst must not close healthy connections; a handshake that has not been accepted yet is worth less than the ones already running |
| D-112 | 2026-11-02 | Workers borrow the shared state (no `Arc` clones on the data path), `WorkerShared` and each inbox are aligned to 128 bytes, read-then-write replaces unconditional writes on the per-turn flags, and work that finished at once runs up to 16 rounds before the next look at the OS | a cloned `Arc` was a contended atomic on every turn: removing it took B-1 on Linux from 561k to 795k round trips per second and doubled Windows' single-connection rate. No security impact: kills, commands, and the inbox are still seen within 16 rounds, about 100 us [est.] under full load |
| D-113 | 2026-11-02 | A pipe is `Stream::Pipe` on the same connection core. On the blocking pool it emulates a non-blocking socket with overlapped IO against 16 KiB buffers made on first use, and waits on its event and the wake channel's, which gains an event. On the reactor those buffers never exist. `as_sock` hands out the handle's number, so socket options fail with `WSAENOTSOCK` rather than needing a second code path. A pipe has no linger | one set of limits, deadlines, kill, and gate for every transport (D-093's reasoning). Cost: one `SetEvent` per wake on Windows; 32 KiB per blocking pipe connection |
| D-114 | 2026-11-02 | `PipeSecurity::OwnerOnly` is a protected DACL naming the current user's SID, not the `OW` alias | under an elevated token a new object's owner is the Administrators group, so `OW` would lock out the same user unelevated. Cost: one token query per listener |
| D-115 | 2026-11-02 | AF_UNIX on Windows runs as `Stream::Tcp` over the Winsock socket, in its own `local_windows.rs`. `LocalOptions`' mode, owner, group, and `require_uid` are refused with `Unsupported` | a socket is a socket to Winsock, so no new IO path; an option that would silently do nothing is a security bug. AF_UNIX and pipe listeners cannot be handed off (register R-23) |
| D-116 | 2026-11-02 | The Windows hand-off frame is the manifest, a 16-byte nonce, one `WSAPROTOCOL_INFOW` per listener, and a CRC32C. The nonce table lives for the process, and the receiver checks the pid at the far end of the pipe, whichever end it holds | Windows returns the same socket for a description used twice (S-1), so the check must come before `WSASocketW` (edge case 104). Cost: 16 bytes per hand-off held forever, a handful in a process's life |
| D-117 | 2026-11-02 | Happy Eyeballs is one `poll` loop over at most 16 attempts in `connect_racing`, IPv6 first and interleaved, at least 10 ms apart; `attempt_delay` is clamped to 100 ms through 2 s instead of failing; losers are reset | no thread per attempt, and `connect_host`'s plain path is untouched |
| D-118 | 2026-11-02 | FreeBSD shares macOS's branch of `sys/unix.rs` (`not(target_os = "linux")`) with its own constants, and runs in `vmactions/freebsd-vm` (D-035). `peer_credentials` reads the pid from `LOCAL_PEERCRED` | the two BSD interfaces differ in numbers, not shape |
| D-119 | 2026-11-02 | The std-only fallback (D-064) moves to 0.6. The design holds: blocking reads under `SO_RCVTIMEO` through `std`, a kill through `shutdown` by raw descriptor, socket options reported `Unsupported`, reactor off. It needs a cfg alias across about 70 `unix` sites (a build script), and a `net_io_fallback` cfg so it runs on Linux in CI. No consumer needs another target today | every supported target works, and the others fail at compile time with a message that says when. The work is real, and it must not land untested |
| D-120 | 2026-11-02 | The cfg `net_io_force_blocking` makes `Auto` resolve to the pool, and CI runs every suite that way on every OS | `Auto` now means the reactor, so without this the pool's session path would lose its coverage |
| D-121 | 2026-11-02 | `new_reactor_driver` returns `Box<dyn Driver>`; the concrete types stay private. `Operation::Close { abortive }` now sets linger 0 on every driver | the interface is Tier 3 and unstable; the box costs nothing that matters at creation |
| D-122 | 2026-11-02 | `defer_accept`, `fast_open`, `incoming_cpu`, and `user_timeout` follow the option table; fast open is set after `listen`, as every OS accepts. Options an OS lacks fail with `Unsupported`, never silently | fast open trades one round trip for replayable first data, so it is off by default and the doc says so; `defer_accept` costs a server that speaks first its greeting, and saves an accept for a silent client |
| D-123 | 2026-11-02 | A file is an output chunk (`Chunk::File`) that travels by its own operation (`Operation::SendFile`): `sendfile` on Unix, `TransmitFile` on IOCP. The gate checks 1 MiB at a time. TLS, pipes, Unix sockets off Linux, and the blocking path on Windows read in pieces. The read happens on the worker | zero copy where the OS has it, one gate seam for every byte. Cost: a slow disk holds a worker on the read paths and on page-cache misses in `sendfile`; `sendfile` flips a blocking Linux socket to non-blocking and back, two system calls per file |
| D-124 | 2026-11-02 | A file that ends early resets the connection with `Io(FileTruncated)`; a `sendfile` that turns the file down before any byte went falls back to reading and writing | the peer was promised a length, and bytes already sent cannot be taken back |
| D-125 | 2026-11-02 | `recv_batch` and `send_batch` use `recvmmsg` and `sendmmsg` on Linux, a loop elsewhere, 64 per call. Budgets run per datagram after the batch arrives, so a dropped one leaves its slot `None`; the guard charges before the send, so a short send understates a peer's allowance and never overstates it | the guard's rule holds on every path. Cost: 8 KiB of stack per batch |
| D-126 | 2026-11-02 | Packet info is Unix only (`recvmsg` with ancillary data); on Windows `bind_with` says `Unsupported`, because it needs `WSARecvMsg` through an extension pointer and no consumer asks. Multicast joins go through `std` | no FFI where `std` has it |
| D-127 | 2026-11-02 | A UDP listener is a thread of its own, not a reactor worker: the thread owns the socket, so the guard stays lock-free and the session needs no synchronization. The door, stop, and drain apply as on TCP. `ServerBuilder::udp_sessions` on a builder with no TCP work makes a `UdpOnly` server with one idle worker | sharing a socket across workers would put a lock in front of the source table on every datagram. Cost: one core per listener; `API.md` said reactor workers, and now says this |
| D-128 | 2026-11-02 | `Pressure` shrinks handshake and idle deadlines by a per-connection scale, floor 1 s, with hysteresis between `high` and `low`. The reactor walks its connections once on each crossing; the pool reads the scale before each wait | no cost below the threshold beyond one check per connection opened or closed. A crossing costs one wheel reschedule per connection |
| D-129 | 2026-11-02 | `lane_reserve`: a connection in lane `l` is refused with `LaneReserve` when the open count plus the slots reserved for lanes before `l` passes `max_connections`. It runs after the gate, since only the gate knows the lane | conservative: slots an important lane already uses still count as reserved. Per-lane counts would be exact, at one more contended counter per admission [est. 10 to 40 ns] |
| D-130 | 2026-11-02 | `Placement` pins workers (`sched_setaffinity`, `cpuset_setaffinity`, `SetThreadGroupAffinity`); `PinPerCore` is CPU `n` for worker `n`. A failed pin leaves the worker where it is | tuning must not stop a server. Physical cores before SMT siblings needs a topology read, moved to the 0.9.x tuning pass (register R-28) |
| D-131 | 2026-11-02 | `ListenOptions::per_worker`: one `SO_REUSEPORT` socket and accept thread per reactor worker, Linux only, each feeding its own worker through a thread-local, and with `Placement` steered by `SO_INCOMING_CPU`. It cannot be handed off | D-104's accept threads stay the default. `EPOLLEXCLUSIVE` in each worker's epoll was not built: this reaches the same end with no second accept path in the loop (R-19) |
| D-132 | 2026-11-02 | The std-only fallback (D-064, D-119) moves after 1.0 as P3. Without `poll`, `std` offers no wait on a socket and a wake together, so a fallback would sleep-poll accepts and writes, which breaks the kill and deadline bounds this crate promises. It also needs a build script and a cfg alias across about 70 sites. A new target is better served by declaring its constants | every supported target works; the others fail at compile time with a message that says so |
| D-133 | 2026-11-02 | `compare/` is an unpublished crate with its own workspace. A closed-loop generator shared by every stack, each server in its own process. Open-loop timing as planned in 9.2 is not built: closed loop measures capacity, and p99 here understates latency under overload (coordinated omission) | one number per shape for the 1.2x gate; the open-loop generator moves to the 0.9.x pass (register R-29). Stacks: `std`, `mio`, `tokio`, `monoio`. `glommio` and `compio` are not in: `monoio` stands for io_uring |
| D-134 | 2026-11-02 | An `Output` keeps one free buffer between sends and gives the rest to the worker's pool the moment a send completes, with four held at most. Before this, a buffer moved in with `push_buffer` stayed on the output forever, so an echo that swaps its input allocated one 16 KiB buffer per round trip, with about five page faults each: 11,002 round trips/s where the copying echo did 560,235, and the rate stopped growing with workers (24,241 on one, 25,130 on two), which points at contention in the kernel's fault path. The zero-allocation test now runs on every driver and on the moved-buffer path; it missed this because it covered only the default driver and the copying path. | The cost of returning a buffer is one `Vec` push and pop per send. Keeping none would make every copying response take from the budget, which is a contended atomic. |
| D-135 | 2026-11-02 | On kqueue the write filter exists only while a send or connect waits, one event at a time (`EV_DISPATCH`), added through `WriteArm`, a copyable handle usable while a wait delivers events. Registered for good it fired after ACKs: 3.05 to 3.13 system calls per round trip on macOS against 3, which closes R-24. Adding the filter evaluates the socket, so a send that blocked just before the buffer drained still gets its event. | The cost moves to the rare case: one more `kevent` per blocked send, against an empty wake on about one round trip in ten. A failed add waits for the write deadline. |
| D-136 | 2026-11-03 | The accept storm test (T-19) stops at 3,000 connections per driver, and every CI job has a timeout of about five times its usual run. Unbounded, the storm used every ephemeral port: closed client ports wait out TIME_WAIT, so the next connect in the run failed with `EADDRNOTAVAIL` on macOS, and tests whose helper thread connects then waited in `accept` until GitHub's six-hour limit on macOS and Windows | 3,000 a driver is still a burst of connects within the second, while a round trip on an established connection is timed, which is what T-19 tests; it leaves most of a 16,384-port range free. A hung job now fails in under an hour |
| D-137 | 2026-11-03 | On the blocking pool, with `Pressure` set, an idle wait lasts at most the idle deadline pressure would give (idle timeout times `factor`, floor 1 s); when it ends without data the read returns `Woken`, the engine reads the scale again, and the deadline is computed again from when the connection went idle. D-128 read the scale only before each wait, so a connection already waiting when the server crossed `high` kept its full idle timeout: CI on Windows showed 2 of 6 closed, 4 still waiting on 30 s. The test now makes that order every time | the pool has no way to reach a thread blocked in `recv` short of a wake per connection. Cost: one extra wait (and on Linux a `setsockopt` when the timeout changes) per idle pool connection per shrunk deadline, at least 1 s apart, only when `Pressure` is set; a pool holds hundreds of connections, not tens of thousands. The reactor is unchanged |
| D-138 | 2026-11-03 | The reactor checks the minimum rate on every arrival and every send completion, as well as on its timer. With the timer alone, a byte arriving after a check fell due but before the next turn fired the timer re-armed it from that moment, so a peer sending more often than `check` (1 s by default) was never measured: in 0.5.0 a client sending a byte every 40 ms after an opening burst held a reactor connection for as long as it kept sending, and each byte also moved the read deadline. The pool always checked before each wait and was not affected. `server::test_trickle_faster_than_the_check_is_closed` fails on the old reactor and passes on both drivers. The output side has the same shape and gets the same check; a slow reader did not starve the write timer on Linux, so that half is a guard, not a measured fix | one `too_slow` per receive and per send: it ages at most ten counters and sums them [est. 10 to 20 ns, against a system call of about 1 us]. Anchoring the timer to fixed times instead was tried and rejected: completions run before timers in a turn, so a byte inside the firing delay still skips each check |
| D-139 | 2026-11-04 | James's decisions for 0.6.0. (1) The 1.2x gate miss (R-30) is accepted for 0.6.0 with the numbers measured on GitHub's 4-vCPU runner and the 32-thread build host (`docs/PERFORMANCE.md`). Re-running the gate on a dedicated, quiet load machine is a required item of the 0.9.x polish milestone; before 1.0 the gate is met, or the target is derived again with the reason written down. (2) OD-26: no FastCGI on Windows before 1.0. (3) 0.5.0 is yanked once 0.6.0 is on crates.io and docs.rs, for the reactor slowloris issue fixed by D-138 | a yank only stops new lockfiles from choosing 0.5.0 and can be undone; the gate decision costs nothing now and keeps the 1.0 target open |
| D-140 | 2026-11-04 | A database consumer's three transport needs are placed by where they fit: bulk transfer that cannot starve small requests and an outbound client that survives its server are 0.7 hardening; tail latency through the `executor` adapter is 0.8, with the adapter. The client work is reconnect with backoff and jitter, a health probe, and a client-side drain on `TcpConnection`; a connection pool stays out of net-io (section 3: `pool-mod`, `fetch-net`), and the soak drives the client through a small pool in the test. Receiving into a file is added because `splice` is the one direction `sendfile` does not cover on Linux | each piece is a transport concern a database hits first (replication streams, snapshot shipping, a replica that must find its primary again); a pool inside net-io would duplicate the pooling crates' policy. Costs are stated with each piece when it lands |
| D-141 | 2026-11-04 | The reactor sends at most 256 KiB of file between two looks at the OS, in pieces of 256 KiB (one gate check each): a connection whose next piece would pass that waits on a list, the worker polls (the readiness driver no longer skips that poll while local completions are queued), and the list resumes first, oldest first. Before, a file to a fast reader on Linux could run 16 local rounds times 16 skipped polls of 1 MiB `sendfile` calls, 256 MiB, before the worker looked at its other sockets, because `sendfile` copies in the worker's own thread. `shim::test_file_stream_looks_at_the_os_every_256_kib` fails on 0.6.0 and passes now (264 waits for 64 MiB). Measured on the loaded build host (WSL2, one worker, a 4 GiB stream to a fast reader, 64-byte pings beside it, three runs each): 0.6.0 p99 258 to 305 us, max 22 to 48 ms; now p99 213 to 216 us, max 17 to 32 ms; stream 2.0 to 2.3 GiB/s against 1.6 to 2.8 before, within the host's noise. p99.9 is the host's own noise there (quiet p99.9 0.09 to 10 ms) and is measured again on a quiet machine at 0.9.x | 1 MiB pieces with the same bound were tried: p99 390 to 447 us, since one piece is the wait. Cost: four `sendfile` calls and up to four extra zero-timeout polls per MiB, about 2 us against about 80 us of copying per piece [est. under 3%]. Windows: `TransmitFile` copies in the kernel, not the worker, so the bound only adds the poll there. Security impact: none; the gate is asked more often, never less |
| D-142 | 2026-11-04 | Client additions: `Backoff` (100 ms doubling to 30 s, each delay drawn from the upper half of its step, optional attempt and time limits), `TcpConnection::connect_retry` (retries refused, timed out, unreachable, reset, no addresses, and failed resolution; resolves again each attempt, so a failover that moves the name is followed), `probe` (zero-timeout wait plus a one-byte peek, nothing consumed), and `drain` (FIN, discard until the peer's FIN or a deadline, then close) | upper-half jitter rather than full jitter: full jitter can retry at once and a fleet that lost its primary should never hammer it, while half the step still spreads a fleet over a range as wide as the step. The jitter source is a xorshift seeded from `RandomState`, no dependency and no allocation. Cost: `probe` is one or two system calls; `connect_retry` sleeps on the calling thread, bounded by the backoff's limits |
| D-143 | 2026-11-04 | Review finding (critical, affects 0.5.0 and 0.6.0): on the reactor, bytes read with a PROXY header on a TLS listener were delivered to the session as plaintext once the handshake finished (and a ClientHello in the header's segment stalled the handshake). `Link::attach_tls` now moves read-ahead into the engine's ciphertext input and returns the emptied buffer; the reactor never delivers read-ahead under TLS. The blocking driver already fed it to the engine | unauthenticated bytes attributed to the TLS peer are the STARTTLS injection class. Cost: one copy of what was read ahead, once per connection. `tls::test_proxy_then_tls_on_sessions_feeds_read_ahead_to_the_engine` failed before |
| D-144 | 2026-11-04 | Review finding (high): `Input` records when a receive holds its buffer (`lend`, `give_back`); while lent, `bytes()` is empty and safe, `is_full` is false, and `release_idle`, the sweeps, and the growth path leave it alone; `Step::Read` waits for the receive. `BufferPool::put` drops a stand-in that owns no memory | a session that survived a read timeout with a partial message held made the worker index a zero-length stand-in and panic, taking every connection on that worker. Cost: one flag. `server::test_session_survives_a_timeout_with_a_partial_message_held` hung before |
| D-145 | 2026-11-04 | Review finding (high): every `TlsAcceptor` and `TlsSession` call runs behind a panic boundary at the seam (a panic becomes `TlsFailed` and poisons the session), every event sink call is guarded, and each worker has a last boundary: the pool returns a job's caps, `busy`, and close when it ends without `end` (also when `set_gate` took the link), and the reactor closes every connection with its accounting | the engine parses the most hostile bytes; one panic cost a worker, its cap slots, and every later drain. Cost: none unless a panic unwinds |
| D-146 | 2026-11-04 | Review findings: the accept batch re-reads the door, a drain, a stop, and the control line's epoch before each accept; a stop (or `Stopping` on the control line) during a drain kills at once; the deadline kill repeats every pass; a job taken while stopping closes unread; one drain emits one `DrainFinished` | up to a batch of connections got in after the door closed, a late one could outlive the deadline kill, and `Stopping` after `Draining` waited for the drain's deadline. Cost: four atomic loads per accept against a system call |
| D-147 | 2026-11-04 | Review findings: the global cap is the server's (`Shared::max_open`); a listener setting `max_connections` is `InvalidConfig` (it caps itself with `max_per_listener`); `max_connections` is at most 2^20; the source table reserves at most 8,192 entries a shard up front | a listener could lift the global cap and the descriptor budget checked against it, and a huge cap aborted the process at build. Cost: none |
| D-148 | 2026-11-04 | Review findings on TLS: a denial is encrypted into a record and sent once (or not at all); records the engine sends on its own neither end an exchange nor escape the outbound rate; a blocked send honours the handshake deadline; on the reactor any ciphertext starts an exchange, as on the pool | the denial went out as plaintext between records; KeyUpdate-style records reset the slowloris grace; a handshake flight drained a byte at a time held a pool worker past the handshake deadline; empty records moved the idle deadline for ever. Cost: one encryption per denial; none elsewhere |
| D-149 | 2026-11-04 | Review finding: a buffer keeps the longest length it held since its last wipe, and the wipe covers that much | `clear`, `compact`, and a consumed input shrank the length first, so the reactor returned buffers with whole requests and decrypted plaintext in them. Cost: one `max` per `set_len`; a wipe costs what the buffer actually held (127 ns per 16 KiB) instead of what it held at release, which was often nothing |
| D-150 | 2026-11-04 | Review finding: timeouts over a year (and the minimum rate's durations, and the linger) are `InvalidConfig`; durations at call sites (`set_idle_timeout`, `accept_timeout`, `ConnectOptions::timeout`) are held to a year | `Instant + Duration::MAX` panicked on the first deadline, outside any boundary. Refusing at build says so; holding at a call site keeps a "no limit" idiom working. Cost: none |
| D-151 | 2026-11-04 | Review findings on the reactor: a paused reader whose wait moved on is resumed at its next read; a failed OS wait closes every connection at once with its accounting; the completer's wake flag is cleared with a swap; a flush behind the engine's records keeps its timer and continues after them; a linger that began with a receive out starts its read when it returns; a PROXY buffer with nothing after the header goes back at once; the wheel holds a clock that went back at its own time; running out of local rounds makes the next wait poll | each could stall or abandon a connection, pin a buffer, lose a wake for up to a park deadline, or (the wheel) spin a worker. Cost: none measurable |
| D-152 | 2026-11-04 | Review findings, low: `Accepted` and `Admitted` come from the worker that takes the job, before anything it does; drains and sheds read a worker's generation before checking whose connection it is; coalesced floods flush when the accept loop sleeps; 64 accept errors in a row with no connection pause the loop; a bad lane is a separate flag; one gate offer is at most `u32::MAX` bytes; a refused `sendfile` resets the gate's approval | event order, a drain hitting the next connection, a flood's tail held for hours, an accept thread spinning on `WSAENETDOWN`, an invalid option silently accepted, and gate counts off by up to a GiB. Cost: none measurable |
| D-153 | 2026-11-04 | Found by the new `rate` fuzz target: the requirement is rounded up, and a counter has at least two buckets | rounding down required nothing at 1 B/s over 200 ms; with a window under twice the check interval, one bucket covered nothing at its own boundary, which is when the timer's check runs, so a silent peer was never too slow there. Cost: none |
| D-154 | 2026-11-04 | A pipe listener counts its open instances under one lock and passes `FILE_FLAG_FIRST_PIPE_INSTANCE` whenever it holds none; `accept` reports `PipeNameInUse` when that fails | a pipe exists only while an instance is open; without the flag a later instance joined a pipe another user created in the gap, with that user's DACL. Cost: one uncontended mutex per accept and per close [est. under 1 us] |
| D-155 | 2026-11-04 | `PipeConnection::server_pid` and `server_owner_sid` (the owner as a canonical SID string, from `GetSecurityInfo`); the caller compares | any local user can create an unheld pipe name; the owner of the pipe reached tells a squatter from the real server with no race. Opt-in, one system call each |
| D-156 | 2026-11-04 | On Windows a drain wakes an idle owner through its wake channel only; `CancelIoEx` stays for an owner with none | `CancelIoEx(NULL)` also cancelled a pipe reply already reported as sent. Cost: one system call fewer per drain |
| D-157 | 2026-11-04 | Hand-off on Windows: every description is claimed before any is verified, and all are closed on a failure; a process-wide list of adopted handle values refuses a second adoption; sockets made from descriptions are made non-inheritable | unclaimed descriptions leaked listening duplicates, a repeated description gave one handle two owners, and `WSA_FLAG_NO_HANDLE_INHERIT` does not reach such sockets (observed on Windows 11). Fails closed: a later hand-off that reuses a reserved value is refused until the engine releases values on close (register R-35). Cost: one mutex and a short scan per adopted listener |
| D-158 | 2026-11-04 | On Windows, a stale AF_UNIX path is removed only when its reparse tag is `IO_REPARSE_TAG_AF_UNIX`; `TransmitFile` past the end of a file fails, so a success always moved `len` bytes (no change, test added) | every reparse kind sets the attribute; cloud placeholders and app aliases were deleted. Cost: one open and query per bind on an existing path |
| D-159 | 2026-11-04 | `Driver::register` takes a `SharedSocket` (an `Arc` of the socket) and holds it until `deregister`; `BorrowedSocket` leaves the public API (Tier 3, unstable until 1.0); the server's reactor keeps a crate-internal borrowed registration | the driver did IO by descriptor number, so safe code that dropped a registered socket let a later connection reuse the number and receive another's bytes; an `unsafe fn` would put `unsafe` outside `sys/`. Cost: one `Arc` per registered socket, nothing per operation |
| D-160 | 2026-11-04 | UDP: verification is per address and port, lasts `verification_lifetime` (60 s, 1 s to 1 day) from the latest `mark_verified`, and lives in its own bounded table (`verified_table`, 65,536); `forget_verified` ends it; only an admitted datagram counts as use in the source table, and a full table's own traffic turns the hand; `amplification_window` is at most an hour | a verified IP was an unlimited reflection target on every port for ever, verified entries could fill the table and lock out newcomers, and spoofed traffic kept every entry in use. Cost: one more lookup per unverified send [est. under 30 ns], about 100 bytes per verified peer [est.]; a protocol must prove again at least every lifetime |
| D-161 | 2026-11-04 | Unix: a cut `SCM_RIGHTS` message closes every descriptor it carried (2,064-byte control buffer, more than 64 descriptors refused like a truncation); FreeBSD sets close-on-exec atomically; a systemd or FastCGI descriptor with close-on-exec set is not inherited (it cannot have crossed `execve`), and a failed claim ends the read for good; a sub-second linger rounds up to a second on every OS; abstract Unix listeners are identified by their full name; flow labels are copied as `std` does; every FFI struct has a size and alignment assertion | each closes a leak, a wrong owner, or a reset the caller did not ask for. Cost: 2 KiB of stack per descriptor receive (a cold path), one `fcntl` per inherited descriptor at start-up |
| D-162 | 2026-11-04 | 0.7 test additions: stateful fuzz targets `wheel` (against a map of deadlines) and `rate` (its invariants), run 30 s per push and 10 minutes nightly with the corpus kept (`.github/workflows/fuzz.yml`); `tests/soak.rs` (normal, pooled, and hostile peers with memory, handle, and connection sampling, `NET_IO_SOAK_SECS`); attack tests for T-14 on the wire, T-22, and T-40 with a real `fork` | the 0.7 exit asks for continuous fuzzing, an attack test per threat, and a soak; the `rate` target found D-153 within a minute |
| D-163 | 2026-11-04 | 0.6.1: a post reaches a connection waiting for input as well as a parked one, on both drivers (the reactor delivers it in a read wait and holds one for a busy session until its next wait; the pool takes it from the ring when the read wait is woken); the consumer clears the wake flag with a swap; on the pool on Linux, a connection whose session took a completer waits on its wake channel before each receive (`poll`, then `recv`), as Windows and macOS always do | serve-net request 7: a WebSocket waits for client input and must be writable from other threads, and a Linux pool receive blocked under `SO_RCVTIMEO` cannot be woken. Cost: one `poll` per receive on the Linux pool, only for connections that took a completer; none elsewhere |
| D-164 | 2026-11-04 | 0.6.1: a step that ends in `Read` with a receive already out releases the connection's idle buffers; `release_idle` leaves an output alone while a send holds its chunks | serve-net request 4: a trimmed idle connection kept a 16 KiB output until its next request (4.9 KB of heap each, 1.8 KB now). Cost: none |
| D-165 | 2026-11-04 | 0.6.1: `close_all` (a stop, a deadline kill) also ends a connection's linger | a linger timed on an injected clock that does not move held `shutdown` for ever (serve-net 0.5 item 2); a single kill already ended a linger. Cost: none |
| D-166 | 2026-11-04 | 0.6.1 carries three fixes from the 0.7 review because they affect 0.6.0 users now: read-ahead after a PROXY header goes to the TLS engine (D-143), a lent input buffer is never read or released (D-144), and the accept batch re-reads the door before each accept (part of D-146). The rest of D-146 and D-147 change behaviour a configuration may rely on and wait for 0.7 | a patch release is where a security fix belongs; the D-143 bug is critical for anyone behind PROXY with TLS on the reactor |
| D-167 | 2026-11-04 | 0.6.1: a drain closes a connection between exchanges only when nothing waits in its socket (one zero-timeout wait on the pool, the readiness flag on the reactor); a request that already arrived is read and answered, and the drain closes the connection after it | serve-net's drain race, second window: a drain pass could close a connection whose first request was in the kernel but unread, and a client then saw a reset after 0 bytes (seen under load in the Windows hand-off test). The drain deadline still bounds a client that keeps sending. Cost: one zero-timeout wait per drained pool connection |
| D-168 | 2026-11-04 | `net_io::descriptor_limit()` is public: the soft `RLIMIT_NOFILE` on Unix, `None` when unlimited and on Windows | serve-net leaves the reactor's global cap unset so a low limit does not refuse to start, and needs the limit to place its own soft cap (and a 503) under the hard cap net-io picks. Cost: none |
| D-169 | 2026-11-04 | 0.6.1: an output takes the buffers its worker's pool already holds (up to four) before it charges the budget, each time a session reaches it; a flush that finds nothing queued after a push the budget refused closes the connection with `Limit(BufferBudget)` instead of calling the session back | ws-net's soak: spares went to the pool after each send (D-134) and outputs never took them back, so every flush of more than one buffer charged the budget again, about 220 MB/s at 50 connections; once it was spent, a session that returned `Write` with nothing taken was called back at once, about 30 million times a second, and every connection stalled. Closing is what a read on an empty budget already does at setup (PROXY, TLS); the leak fix keeps a normal load far from it. Cost: up to four pops per output reach, no allocation; `server::test_multi_buffer_flushes_reuse_their_buffers` and `server::test_spent_budget_closes_instead_of_spinning` failed before |
| D-170 | 2026-11-05 | `TcpConnection::recv_file(file, offset, len)`: receives up to `len` bytes into a file at an offset, short only at the peer's FIN. On Linux each receive is a `splice` from the socket into a pipe and from the pipe into the file, run as the receive call inside the link's read path, so the read deadlines, the minimum rate, kills, and drains apply unchanged; bytes read ahead and TLS go through a 64 KiB copy, as does every receive after the socket or the file turns `splice` down. Bytes already in the pipe when the file refuses are read out of it and written, so a refusal never loses data. Elsewhere it copies (`pread`/`pwrite` style, no position change on Unix) | the receiving half of snapshot and log shipping (D-140); `sendfile` covers only the sending half. Measured on the loaded build host (WSL2, 1 GiB over loopback, three rounds): 1.6 to 2.6 GiB/s against 1.0 to 3.0 GiB/s for `recv` plus a positioned write, within that host's noise; the saving is the copy through user space, about one memory copy per byte [est.], measured again on a quiet machine at 0.9.x. Cost: one pipe (two descriptors) for the length of the call, which a handler that calls it must leave room for under the descriptor limit; one extra `splice` per receive. Security impact: none on the limits, which run as for `recv`; `len` bounds what a peer can write |
| D-171 | 2026-11-05 | Fuzz target `udp`: datagrams, sends, verifications, forgets, and clock steps against the UDP guard with tables of 1 to 16 sources and 1 to 8 verified peers, checking table consistency and bounds, no verification a model does not hold (expiry and `forget_verified` included), no send to an unverified peer past the factor times what its entry holds as received, and the global budget's burst plus rate. It also runs 2,000 pseudo-random inputs as a unit test on every platform | the 0.7 fuzz plan names the UDP source table; the guard is what stands between a spoofed source and an amplification. 2.46 million runs in 5 minutes found nothing. Cost: CI time only |
| D-172 | 2026-11-05 | James's method for the 0.9.x speed gate: the load generator and the server under test run on separate pinned core sets on the 32-thread build host in a quiet window, beside the hosted-runner numbers; a run with the generator on its own machine goes into the 1.0 readiness note as an open item. Under his standing order, technical, scope, and sequencing calls are made here and recorded as D-entries; only 1.0, money or hardware, filings outside the repository, and visibility go to him | pinning removes the shared-CPU contention that made the 0.6.0 numbers uncertain (D-139) without new hardware; no machine is bought for it. Cost: a quiet window on the build host, which up to ten build sessions share |
| D-173 | 2026-11-05 | A thread's wake channel (the one a `TcpConnection` used outside a worker waits on) is never closed when its thread ends: the thread-local's destructor gives it to a process-wide list of spares, and the next thread that needs a wake takes one from there. A poisoned list leaks the wake rather than close it. Windows only: elsewhere a thread-local destructor may close a descriptor, and keeping wakes held two descriptors per ended thread on macOS (the soak's descriptor check caught it) | on Windows a thread-local destructor runs under the loader lock, and closing a socket there is the documented deadlock (no Winsock calls from `DllMain`): with a thread loading a Winsock extension (the IOCP and `TransmitFile` tests were in flight) the whole process froze, every thread start and exit waiting. The library tests on Windows hung in about 1 run in 14 with tests in parallel, once for 30 hours. Cost: one uncontended mutex at a thread's first wait and at its end; the list holds at most as many wakes as threads ever waited at once (a socket and an event each on Windows). `link::test_thread_wake_is_kept_when_its_thread_ends` fails before |
| D-174 | 2026-11-06 | The reactor's hand-over looks for the least-loaded worker from a start that moves on by one each connection, and stops at the first idle worker; ties go round the workers. 0.6.2 ships the Windows deadlock fix (D-173) on its own, right after 0.6.1, rather than holding 0.6.1 for it | tls-net's handshake benchmark: with short connections every worker's load is usually 0, and taking the first minimum handed every connection to worker 0, so one core ran at 100% (3.5 to 5k handshakes/s on 8 workers; 379 MB/s of bulk where 7.6 GB/s was possible). `server::test_short_connections_spread_over_reactor_workers` used 1 of 4 workers before. Its linger report did not reproduce on Linux with the toy TLS engine under 64 clients (peak open 95 with the default linger, 88 with none), so the pile-up there is put down to the one busy worker until tls-net re-runs; the test written for it found a Windows defect, D-177. Cost: one relaxed atomic add per accepted connection, and the scan stops early. Security impact: none; the per-listener `FEEDS` path is unchanged |
| D-175 | 2026-11-06 | On Windows a `listen` backlog past 200 is passed as `SOMAXCONN_HINT(n)` (the negative of `n`), held to 65,535, and `ListenerInfo::backlog` reports it; 200 or fewer stay a plain number | Windows silently takes a plain backlog as at most 200: the default 1,024 queued exactly 200 connections here, and the next connect waited for its SYN-ACK until it timed out. ws-net's churn soak on a loaded host (about 99% CPU from other sessions) saw 130 of 200 clients fail together with `WSAETIMEDOUT` while the listener still accepted others. `sys::windows::layout::test_backlog_past_two_hundred_is_kept` queued 200 of 300 before. Cost: kernel memory for the pending connections a larger queue holds, which the backlog already promised; no system call added. Security impact: a SYN flood gets a longer queue to fill, which SYN cookies (on by default on Windows) cover |
| D-176 | 2026-11-06 | On Unix a drain checks the idle connection for waiting input (one zero-timeout `poll`, under the same guard as the shutdown) before it shuts the read side; with input waiting it takes its mark back, the owner reads and answers the request, and a later pass closes the connection at its next wait | on the Linux pool a receive blocks without a wait first, so D-167's check did not run there: a drain that found a connection idle with its next request queued shut the read side, the queued bytes still arrived, and every later read in that exchange was end of stream, closing a handler mid-exchange with `Drained` (CI: `admission::test_stopping_during_a_drain_kills_at_once`, Linux, blocking driver forced). A shut read side cannot be opened again, so the check must come before it. `sys::kill::test_drain_leaves_a_connection_with_input_waiting` fails before. A request that arrives between the `poll` and the shutdown still loses its later segments: a window of one system call, against the whole idle wait before. Cost: one `poll` per idle connection per drain pass |
| D-177 | 2026-11-06 | On the reactor, a receive that fails during the linger (the peer's reset) ends the linger at once; it no longer starts another read | on Windows the receive after a reset does not complete, so a client that reset after reading its reply (`abort`, or closing with bytes unread, which is how a TLS client that skips `close_notify` ends) held the server's close for the full linger, 2 s by default, with its buffers and cap slot. Found by the test written for tls-net's linger report; their run was on Linux, where the read after a reset returns end of stream and this does not happen, so their pile-up there stays put down to the one busy worker (D-174) until they re-run. `server::test_close_after_reply_ends_with_a_reset` waited 2.006 s for each before; 1 to 2 ms now. Cost: none. Security impact: a resetting peer frees its slot sooner |
| D-178 | 2026-11-06 | A drain reaches an owner that has a wake channel through that channel alone, on every system, and the owner decides at its wait: it closes with `Drained` when nothing waits, and otherwise reads and answers the request, clears the mark, and is marked again by a later pass. On Linux the pool now waits (`poll` on the socket and the wake channel) before an idle-phase receive, so the channel reaches it; reads inside an exchange keep the single blocking receive. Only an owner without a wake channel has its read side shut (with D-176's check) | D-176 narrowed but did not close the race: the drain thread decided from outside, and a pass that landed between the owner's receive taking the bytes and its idle mark clearing shut a connection that was now mid-exchange (seen again on FreeBSD CI). Only the owner sees both its phase and its socket. `admission::test_stopping_during_a_drain_kills_at_once` passed 60 of 60 forced-blocking runs on Linux after. Cost: one `poll` per exchange start on the Linux pool, measured as 3.00 system calls per echo round trip against 2.00 before (B-13; the blocking pool on every other system already paid it) [est. 0.3 to 0.5 us, against a round trip of about 10 us on the pool], none elsewhere. A peek before the receive closes the same window at the same cost; a bounded receive timeout instead would add up to its length to every drain and wake every idle worker that often; any check from the drain's thread leaves the window open. Security impact: none; a drain still closes an idle connection at once, and the drain deadline bounds a client that keeps sending |
| D-179 | 2026-11-06 | Accept threads count themselves from before their door check to the end of an admission (`Shared::admitting`, sequentially consistent with the door), and a drain is not over while the count is above zero | the Windows hand-off test lost one connection in about eight runs: the accept thread had read the door open and was admitting a connection when the old server's drain closed the door and found nothing queued, so the drain ended, the stop began, and the worker that took the job closed it unread (a reset after 0 bytes). Cost: two atomic operations per accept, against its system call |
| D-180 | 2026-11-06 | tls-net's two handshake proposals go to 0.8, beside serve-net's requests: (1) the reactor sends what one turn produced in one send: when a TLS step completes the handshake with records written (the tickets) and the request is already in the input, the records wait for the session's `on_data`, and records, response, and a `close_notify` from `Step::Close` leave together (4 sends a connection to 2); (2) with per-worker listeners, each worker accepts on its own socket in its own loop, with the gate there, so the accept thread's `poll` and the inbox hand-off (one cross-thread wake a connection) go | measured by tls-net on 0.6.1 on the shared build host: at a load of about 130 the reactor does 2,517 handshakes/s against tokio-rustls's 1,739 (1.45x) and 2,072 against 1,680 MB/s of bulk, with 617 against 589 us of server CPU per handshake (1.05x) and 17% less CPU per GB; send system calls are about 25% of the reactor's CPU per handshake and the accept path about 8%, so these two are what would put CPU per handshake ahead too (a first run at load 500 showed 1.46x, mostly contention). Both change the reactor's send and accept paths, which 0.7's hardening has just been through, so they are built and measured with tls-net in 0.8 rather than rushed into 0.7. Security impact of (1): none on the protocol; the records leave microseconds later in the same turn, and a client that sends no request gets its tickets on the next send or at close |
| D-181 | 2026-11-06 | On the reactor, a TLS connection waiting between exchanges with nothing in its buffers (handshake done, all ciphertext consumed, all plaintext delivered, no event pending, nothing sending) gives both TLS buffers back to the pool and waits with a zero-byte receive, as a plaintext connection does; it takes a pair again when bytes arrive (pausing on the starved list if the budget is spent), before a flush (closing with `Limit(BufferBudget)` if it cannot, as D-169), and before a graceful close so `close_notify` can go out (a plain FIN without one) | found by tls-net: `cin` and `cout` (64 KiB each by default) stayed attached from the handshake to the close, so idle TLS connections held 128 KiB of the shared budget each and a default reactor (512 MiB) refused connection 4,097 with `Limit(Connections)`; tls-net measured 25.8 KB per idle connection cold and 102 KB after one 32 KiB page, against 11.4 and 11.7 KB for tokio-rustls. It is also an exhaustion vector: idle TLS peers spent the budget plaintext connections draw from. The engine keeps nothing in the caller's buffers between steps (D-066), so a fresh pair is correct. `tls::test_idle_tls_connections_hold_no_buffers` (100 idle connections on a budget for about 32 pairs) failed before. Cost: two pool takes and puts per TLS exchange that starts from idle, and one zero-byte receive per idle wait, as for plaintext. Security impact: the budget bounds active TLS connections, not open ones |
| D-182 | 2026-11-06 | bouncer-io's two close requests (its readiness note, from its D-122) go to 0.8 as its first items, as one new close mode, flush-then-close: no more input, what is queued goes out, FIN, then reads are discarded until the peer's EOF or a bound, then the socket closes (normally: the kernel resets it only if input arrived unread). (1) After a denial whose bytes went out (`Session::on_denied`, or `Refusal::WithBytes` after a PROXY header), the connection closes this way, bounded by a new `Limits::denial_linger` (default 100 ms, bouncer's own `AfterFlush` deadline; zero keeps the immediate RST); a denial with nothing written still resets. At admission `Refusal::WithBytes` is followed by FIN instead of RST but no linger: the accept thread never waits and reads nothing, so input the peer already sent still turns that close into a reset. `Refusal::Reset` stays an immediate RST, so door refusals keep bouncer's T-8 property. (2) `ServerHandle::close_after_flush(id, reason, within)`: a soft kill (a `SOFT` bit beside `KILLED` in the kill slot, with its bound): reads end, sends go on until the bound (`check_sendable`), then the same close; the reactor sends what was queued through its `FlushClose` wait with the write timer held to the bound; the reason is `Killed(reason)`. A `kill` afterwards upgrades it (clears `SOFT`) and ends any linger with a reset; `kill` alone stays abortive. Only the owner is woken, so its linger can still read the peer's FIN; the exception is a Linux pool worker inside an exchange, whose blocking receive does not wait on the wake channel: its read side is shut as a kill does, and its linger ends at once. TLS sends `close_notify` before the FIN. (3) At bouncer-io's review: soft closes are capped server-wide by `Limits::denial_linger_max` (default a sixteenth of `max_connections`, at least 1; zero turns them off): a connection that finds no place closes with a reset (a soft kill then acts as a kill), so a storm of provoked denials costs at most that many slots for at most the linger each; the drain is bounded by bytes too (`graceful_linger`'s cap, 64 KiB), after which input left unread makes the kernel reset. Costs: 8 bytes on every connection (one pointer to a soft-close record allocated only where a soft close happens; the idle reactor connection's state is 904 bytes, 896 before), one atomic add and subtract per soft close; [est.] a denied connection holds its slot and descriptor up to the bound (100 ms); one more system call (`shutdown`) and the reads that drain the peer. On the blocking pool the worker waits in the linger, so a storm of denials caps that pool at workers / 100 ms denials a second until the cap stops it; the reactor lingers without a thread. To test in 0.8: a pool worker hands the lingering socket to a reaper so no worker waits (hypothesis: denial throughput on the pool stays at today's; method: a denial storm against `workers(4)`, denials/s with and without; cost: one shared reaper thread and a queue). `tests/flush_close.rs` covers both drivers: FIN after a denial, a reset with a zero linger, the linger bounded for a peer that stays, a reset past the cap, the queued output delivered after `close_after_flush`, an idle connection closed at once, the bound for a peer that never reads, and a `kill` ending the linger. Found while testing: Winsock takes a large send of static memory by reference, so on Windows a non-reading peer does not keep output queued in the server; a reactor slot has no wake channel and is never shut by a soft kill (its sink wakes it) | On Windows `closesocket` with `SO_LINGER` 0 discards send-buffer data not yet transmitted, so the formatted denial is lost now and then (bouncer measured 4 in 50 on loopback); a FIN queued behind the data delivers it, and the bound keeps a slow or silent reader from holding the slot. A normal close after the bound instead of an RST: an RST can make the peer's stack discard bytes it has not read yet (Windows does), and the kernel already resets when unread input is left. Not 0.7: it changes both drivers' close paths and the kill slot, and 0.7 goes to its 24-hour soaks as it is; 0.8 starts on it while those run |
| D-183 | 2026-11-06 | `Session::on_drain(cx) -> Step`, provided, default `Close`: called once when a drain reaches a connection between exchanges, where the drain used to close it. `Close` flushes what the hook queued and closes with `Drained` (so the default is the old behaviour, plus a flush of nothing); any other step keeps the connection through the drain: it is no longer counted idle for a drain (the pool stops marking its idle waits, the reactor's sweep and read check skip it), and the session closes it, or the drain deadline does. Both drivers. Connections busy when the drain starts hear it at their next idle wait, as they were closed there before. `BlockingHandler` has no hook and is unchanged | serve-net request 7, the part not done in 0.6.1 (D-163): a WebSocket waiting for client frames was closed by a drain with no chance to send its 1001 Close. Costs: none on connections that never drain; per drained connection one hook call. Three flag bits in one byte on the link (the reactor connection keeps 904 bytes: `auto_phase` moved into the same byte) and one bool on the reactor's connection record, in its padding. Security impact: a kept connection holds its slot until the session closes it or the drain deadline, which already bounds every connection in a drain. `tests/drain_hook.rs` covers the goodbye-then-close, the kept connection the session closes, and the deadline ending a kept one, on both drivers |
| D-184 | 2026-11-06 | `connect_retry` retries a reset during the connect and a shortage of local ports (Windows `WSAEADDRINUSE`, `WSAENOBUFS`, `WSAEADDRNOTAVAIL`; Unix `EADDRNOTAVAIL`, `EADDRINUSE`), under the same backoff; a reset during the connect is `NetError::ConnectionReset`, as `from_os` maps it everywhere else (it was `Os { op: "connect" }`) | the 0.7 soak's pooled client counted 4 stuck exchanges on the Linux pool (the server refusing at its cap, a reset seen during the non-blocking connect) and about 20 on Windows (ports exhausted by two soaks and another project's soak on one host): `connect_retry` returned at once on both, where a pool should wait. Both pass by themselves; the backoff's `give_up_after` still bounds the wait, and an error that does not pass (permission, a bad local address set on purpose with `local_addr`, at worst after the give-up) still ends it. `tcp::tests::test_connect_resets_and_port_shortage_are_reconnectable` fails before. Cost: none. The soaks started on 702c2d1 are restarted on the fix |
| D-185 | 2026-11-06 | The soak's slow reader learns of the server's close from `SO_ERROR` (`TcpConnection::take_error`) each second, not only from a read that fails | on Linux a reset leaves what is already in the receive buffer readable first, and at 64 bytes a second the 64 KiB buffer takes over 1,000 s, so every slow reader counted as held past the 20 s limit although the server had closed it (the server counted 188 slow-peer closes in 60 s); short soaks never reached the limit, so the first 24-hour Linux run found it. A server that does not close sets no error, so a reader the server really holds still counts. Windows and macOS never counted slow readers (their kernels take the whole reply). Test only; cost none |
| D-186 | 2026-11-06 | `ListenerSpec::workers(n)`: on the reactor a listener gets `n` workers of its own, numbered after the server's shared ones in listener order; the hand-off picks the least-loaded worker inside the listener's range (the shared range for every other listener). Refused with `InvalidConfig { field: "workers" }` on the blocking driver, with per-worker listeners, on UDP, and from `add_listener` (workers are spawned at build), and `workers(0)` is `Zero` | serve-net request 6: its `Budget::dedicated_workers` keeps threads for an admin listener on the blocking driver, but on the reactor it could keep only connection seats, so a flood on the public listener still shared the admin connection's worker. Extra workers, not carved out of the shared count, so `workers(n)` keeps meaning what it meant. Cost: a thread and an event loop per dedicated worker; the hand-off's scan is over the listener's range. Security impact: an admin listener's latency no longer depends on the public listeners' load. `tests/listener_workers.rs`: public connections stay on the shared workers and admin ones on theirs, and the admin listener answers within 500 ms while every shared worker is held |
| D-187 | 2026-11-06 | R-43 is not a defect of sharing one listening socket between two servers in a process: the resets after 0 bytes are the new server refusing at its own caps (`GlobalCap`, one connection per worker on the blocking driver, and `SourceCap`, 16 per source by default) while its workers start on a loaded host and the old server's clients move over; the caps stay as D-067 set them. `tests/handoff_in_process.rs` runs serve-net's restart in CI on every system, with headroom over the load (64 workers, a per-source cap of 64), and checks that nothing is refused, reset, or unanswered | reproduced on the build host: 2 of 40 runs failed with 16 workers each; traced, every reset was the new server's `Refused { cause: GlobalCap }` (or `SourceCap` once the workers had room), 11 connections already open 50 ms after it started with 4 clients; with headroom 0 of 60 failed (a client out of local ports retries, as a host's TIME_WAIT is not a refusal). serve-net's test on GitHub's small runners had the defaults, so the slower runner hit the caps. Changing the pool to stop accepting at its cap instead of refusing would trade fast refusals for clients waiting in the backlog; not taken without a consumer asking for it. Cost: none |
| D-188 | 2026-11-06 | serve-net's Windows blocking-pool hang when the old server dies before the new one starts is the new server's start time, not a connection held: a server's drop (and `shutdown`) joins its workers and its accept threads and closes its handle of the socket before it returns, so nothing of the old server accepts after it; connections begun in the gap wait in the backlog until the new server's accept thread runs, and are then answered at once. Start-up stays as it is (workers spawned in a plain loop, no hand-shake with each, the accept thread last); `tests/handoff_in_process.rs` adds the drop-before-start case with a client that writes its request first, on every system | reproduced with serve-net's own test pinned to one core beside a second process running its other restart tests: its new server (about 300 threads on the blocking driver) took 11 s from adopting the socket to opening its door and 3.8 s more to its first accept, then accepted all 13 waiting connections in one batch and answered each in about 1 ms; the requests that failed had waited 6 to 11 s in the backlog against a 5 s client timeout. Unloaded, a 256-worker blocking server opens its door in 19 to 31 ms and answers its first request in 64 to 130 ms on this host. Starting the accept thread before the workers would move connections from the backlog to the queue sooner but not answer them sooner, since each still waits for a worker; not taken. Cost: none |
| D-189 | 2026-11-06 | `Limits::first_request_grace` (default 1 s, zero for the old behaviour): a drain does not count a connection idle while it has received nothing and is within the grace of its accept; once the grace passes a later pass of the drain closes it as before, and the drain deadline still bounds it. Both drivers (the pool's idle check before a read and its woken wait, the reactor's sweep and its idle check). A shed does not wait: the kill slot's new `SHED` mark beside `DRAINED` tells the pool's owner which one marked it, and the reactor's sweep applies the grace only for `Drained` | found by `tests/handoff_in_process.rs` on Linux (1 of 10 runs): a graceful drain closed a connection accepted a moment before the door closed, before its client's request arrived, so the client read end of stream with no reply. A connection that never sent anything was idle by every rule the drain had, though its first request was on its way; Go's `http.Server.Shutdown` gives a new connection 5 s for the same reason. 1 s covers a first request one round trip behind the accept on any common path. Costs: none per connection outside a drain (a check of `bytes_in` and the accept time when a drain looks at an idle connection); in a drain, a connection that stays silent keeps its slot and drain open up to the grace, and on the pool its owner is woken by each 10 ms drain pass meanwhile (about 100 wake-ups a second per such connection [est.]). Security impact: a client that connects and says nothing during a drain holds a slot at most the grace longer, against the drain deadline it already had; the idle timeout and a shed are unchanged. `tests/drain_fresh.rs`: a request written 100 ms after the drain began is answered, a silent connection closes after the grace and not before, a zero grace closes it at once, and a shed closes it at once with a 30 s grace. `listeners::removed_listener_closes_and_drains_only_its_connections` counted a greeted connection that never sent a byte as idle; it now has one exchange first, which is what the test means by idle |
| D-190 | 2026-11-07 | `connect_retry` ends at once on `AddrNotAvailable` that names the `local_addr` it was given: the bind of a local address this host does not have, a configuration error. `AddrNotAvailable` naming the remote address (a connect out of local ports) is still retried | D-184 made `AddrNotAvailable` reconnectable for the port shortage, but the bind of a missing local address fails with the same variant, so `client::test_connect_retry_returns_a_configuration_error_at_once` retried under a `Backoff::new()` with no give-up and hung every CI test job at cf35f4a until the 40-minute timeout. The variant carries the address it failed on, the local one for the bind and the remote one for the connect, so no new variant or error stage is needed. `tcp::tests::test_connect_resets_and_port_shortage_are_reconnectable` now also asserts the bind case. Cost: one comparison per failed attempt |
| D-191 | 2026-11-07 | Each stuck exchange in the soak reports the server's view of that client port (accepted, killed, closed, with the reason and byte counts, and how long ago) and the client's (`take_error`, `probe`, then whether one more byte draws a reset); each held hostile peer prints the second it was held; a thread that sleeps 50 ms reports each overrun of a second or more and the worst in every sample (`stall`) | the first macOS soak (cf35f4a, 19,822 s) counted 10 read timeouts on the reactor in 651 million exchanges, and the earlier cancelled one 7 on the reactor and 1 on the blocking pool; Linux and Windows count none in 24 hours. The line said only `TimedOut { kind: Read }`, which cannot tell a server that kept a connection and stopped answering (a net-io fault) from one the kernel never handed to the server or whose reset it did not deliver. The record is one map entry per client port (at most 65,536) and one per open connection, kept by the soak's event sink. The 0.7.0 Linux soak (cf35f4a) held 11 hostile peers on the reactor and 15 on the pool in the one sample at 25,500 s, both runs at the same wall-clock second and never again, while the Windows soak on the same host held none: a stall of the WSL VM [est.], which the stall line now measures instead of leaving to inference. Cost: a mutex taken per accept, kill, and close event, and one thread waking 20 times a second, in the soak only |
| D-192 | 2026-11-07 | A connection whose session's last bytes went out while it was idle (a reply written in `on_open` before the first read, a push between exchanges) lingers on a graceful close on both drivers, instead of taking the idle shortcut that closes at once (`SPOKE_IDLE`, a bit in the link's flag byte, set or cleared on every send; the TLS engine's own records, close_notify among them, leave it as it was) | serve-net's busy refusal (a 503 written at open, then `Step::Close`) closed with a FIN and no linger, because a connection starts in `Idle`; the client's request, already on its way, reached a closed socket and drew a reset, and Windows then discarded the 503 the client had received: 10053 in 9 of 40 runs on the Windows pool, once in serve-net's CI. `tests/reply_at_open.rs` (the client writes 30 ms after connecting) failed on round 0 on the Windows pool before and passes 40 rounds on both drivers after. serve-net works around it with `set_phase(Sending)`; this makes the workaround unnecessary for every session. The shortcut stays for a connection idle after an exchange that it answered, the common case a drain meets. Cost: no memory (a spare bit; a bool padded the idle reactor connection from 904 to 912 bytes, so it is a bit), one branch and store per send; a server-first connection closed while idle (a greeting nobody answered, a refusal at open) now holds its slot and descriptor until the peer's FIN or the linger bound (`graceful_linger`, 2 s by default, at most 64 KiB read and discarded) [est. a drain of idle server-first connections waits up to the bound]. Security: the linger is bounded in time and bytes as every graceful close is, and a soft-close cap or a kill still cuts it short |
| D-193 | 2026-11-07 | The macOS soak's stuck exchanges are resets that the client's stack dropped, not a server that stopped answering: the soak counts a read timeout on a connection the server killed during that wait as a lost reset, printed and sampled (`resets lost`), allowed on macOS and FreeBSD and a failure elsewhere; a stuck exchange still fails everywhere | D-191's record on the 2-hour macOS run (0e262b7) explained all 7: each was a big writer (the server streams 4 MiB), killed by the soak's killer about 1 ms after its accept with 245,760 to 2,899,968 bytes already sent, closed `Killed(7)` (an abortive close, a reset), and the client, reading at full speed, waited out its 10 s read timeout with no reset or EOF; 6 on the reactor and 1 on the pool, so not the driver. None on Linux or Windows in 24 hours. The explanation (its second half confirmed on 2026-11-08, below): with the client's receive window full, the server's reset carries a sequence number past the zero window (a window probe had advanced it), and BSD stacks drop a reset outside the window (RFC 5961); the client's window update then reaches a closed port, whose reset macOS rate-limits (`net.inet.tcp.rst_rlc_enable`, on by default with a 200 ms bucket; the runner has no `net.inet.icmp.icmplim`), which a soak's churn of aborts exceeds. The server did what a kill asks; only the peer's own read timeout ends its wait, so a client of a killed connection needs one (a net-io client has one, 30 s by default). To confirm (hypothesis: the reset limit is the second half; method: the macOS soak with the closed-port reset limit turned off through the workflow's `sysctl` input (the runner has no `net.inet.icmp.icmplim`; each run lists the limits it has), lost resets per hour against today's 3; cost: one 2-hour CI run). Result: with `net.inet.tcp.rst_rlc_enable=0`, 2 hours on each driver (run 42453989445), 0 lost resets on either, against 25 in 5.5 hours on the reactor with the limiter on (the release run at 4e3a0d0) and 7 in 2 hours at 0e262b7; nothing else changed. A deployment on macOS that kills many connections can turn the limiter off; the soak keeps the default, as a deployment would |
| D-194 | 2026-11-08 | D-180's first half, on the reactor: the records a TLS engine writes as its handshake ends (the tickets) stay at the front of the output buffer when the client's request is already in the input, and leave with the session's first reply; a reply sealed before `Step::Close` takes the last of the output carries `close_notify` in the same buffer, so the close sends no second one. A full exchange with a closing reply goes from 4 sends to 2 (the server's flight, then tickets, reply, and `close_notify`). Held records are sent at once whenever the turn ends without a reply (the machine wants input: a request with no answer), go first in a denial and a `close_notify`, and keep the buffers from going back to the pool while held. The blocking driver is unchanged | Why: send system calls were about 25% of the reactor's CPU per handshake in tls-net's measurement on 0.6.1 (D-180), and two of the four sends carried a few hundred bytes each. Why only then: a client that sent no request may be waiting for nothing from the server, but holding records it never asked about past the end of the turn would gain nothing and delay them; a client that sent its request with its Finished waits for the reply, and the tickets reach it in the same segment. The records are encrypted in sequence, so nothing sent after them may go without them; every path that writes into the output buffer appends after them. Cost: no memory (the same output buffer, already held for the handshake), one length field and one flag in the boxed TLS layer (the connection's size does not change); latency: the tickets arrive with the reply instead of before it, the reply itself no later; CPU: two send calls fewer per connection [est. up to half the send share, about 10% of CPU per handshake; tls-net measures it against tokio-rustls]. A session that parks before replying keeps its TLS buffers until it replies or its park ends (bounded by `park_timeout`). Security impact: none on the protocol (the same records, in order, microseconds later in the same turn); `tests/tls_coalesce.rs` checks the order, the single `close_notify`, and that nothing leaves before the reply. Open: D-180's second half, accepting in each worker's own loop Result (tls-net, 2026-11-08, 5970be9: interleaved loopback, 11 rounds of 3 s, 8 server and 16 client cores, load 20 to 47): server CPU per handshake 448 to 400 µs with the request after Finished and 426 to 384 µs with it (10 to 11% less); sends now level with tokio-rustls (203 against 200 profile samples per 10,000 handshakes), and the reactor is still 1.12 to 1.21x tokio-rustls's CPU per handshake (331 and 342 µs). What is left is mostly the accept path and wake-ups (poll, epoll_wait, and accept4: 96 samples against 23), D-180's second half, then `shutdown` on every close (39 against 15) |
| D-195 | 2026-11-08 | `TlsInfo` carries the SHA-256 of the verified client certificate's public key (`with_peer_spki_sha256`, `peer_spki_sha256`), set by the engine, so a session can tell mutual-TLS peers apart and map each to a principal; it goes into 0.7.0 although the release soaks had started | tls-net's request: it verifies the client certificate (the chain, or a pin by SPKI) but net-io had no way to hand the result to the session, so admission worked and authorization by peer did not. The key's hash and not the certificate: it survives reissue with the same key, it is what a pin names, and a fixed array keeps `TlsInfo` `Copy` with no allocation. Trust: net-io never computes or checks it; the engine sets it only after verifying, never from unverified bytes, and the doc says so. Cost: 33 bytes more in each TLS connection's `TlsInfo` (behind an `Arc`, once per handshake), nothing per record. Why in 0.7.0: the change is additive (`TlsInfo` is `non_exhaustive` with private fields), touches no IO path, and the soak runs no TLS, so the soaked code paths are unchanged; a 0.7.1 for it would cost every consumer another bump |
| D-196 | 2026-11-08 | 0.7.0 is released on the soaks at 4e3a0d0 (the server code that ships; later commits changed only the soak, the workflow, documentation, and D-195's `TlsInfo` field): Linux 24 hours on each driver, Windows 24 hours on the pool, macOS 5.5 hours on each driver, and the Windows reactor's 22.7 hours at cf35f4a, whose server code is the same | Linux: reactor 2.62 billion exchanges and 30,217 kills, pool 8.89 billion and 25,734, no stuck exchange, no held peer, no lost reset; the pool's resident memory did not move in 24 hours (27,964 KiB) and the reactor's rose in 256 KiB steps from 12,040 to 13,832 KiB, the last step 6 hours before the end: the worker pools keep each buffer they allocate, up to the shared budget, so their high-water mark rises with the busiest moment so far and then stops (bounded by `Limits` buffer budget, not by time). macOS: reactor 672 million exchanges, pool 699 million, no stuck exchange, 25 lost resets on the reactor (D-193). Windows pool: 6.72 billion exchanges, 57,673 kills, 51.8 million connections accepted, no stuck exchange, no held peer, no lost reset, no session panic; resident memory 44,080 KiB at 5 minutes, 49,192 KiB at 4 hours, 49,384 KiB from 22 hours to the end (the same high-water shape). Windows reactor not rerun: a second 24-hour Windows run at once doubles the host's loopback TIME_WAIT load that R-44 traced, and running one driver after the other would push the release a day for code the cf35f4a run already covered |
| D-197 | 2026-11-08 | D-180's second half: a per-worker group on the reactor (`ListenOptions::per_worker`, Linux) has no accept threads. Each worker registers its own socket of the group in its own event loop and accepts there: `Operation::Accept` armed while no connection waits, a `Nop` to come back when a batch of `accept_batch` ran out (so a backlog cannot hold the worker past one batch a round), the caps, the control line, and the gate's `admit` as an accept thread runs them (one shared `accept_some`, so the two paths cannot drift), and the admitted connection goes to the worker's own inbox, which it takes in the same turn, with no wake (`FEEDS` carries whether the feeder is the worker itself). A closed door or a drain parks the listener, looked at each turn; descriptor pressure pauses it until a deadline the worker's wait honours. Listeners added or removed at run time move an epoch the workers compare each turn; `remove_listener` (and shutdown) returns only when every worker has dropped its socket (each listener's `Arc` count), so the sockets close before it returns, as a joined accept thread's did (D-188). A control-line drain, which an accept thread runs in place, runs on a thread the first worker that sees it starts, once | tls-net's profile of D-194 (5970be9) put the reactor at 1.12 to 1.21x tokio-rustls's CPU per handshake, with the accept path and wake-ups the largest part left: `poll`, `epoll_wait`, and `accept4` 96 samples per 10,000 handshakes against 23. This removes the accept thread's `poll` and its `accept4` wake, the eventfd write that woke the worker, and the worker's extra `epoll_wait` return for it: per connection, about one `poll`, one `write` to an eventfd, one `epoll_wait`, and one thread switch [est.; tls-net measures]. Costs: a few atomic loads a turn for each listener a worker holds (its door, and the epoch); one slot index per listener; the gate's `admit` now runs on the worker, so a slow gate delays that worker's other connections (an accept thread used to absorb it; documented on `per_worker`); a removal waits for every worker's next turn, which a worker held in a long session callback delays, as a gate callback delayed an accept thread's join. Security: none changes; the same admission code runs in the same order (caps, lockdown, gate, lane reserve, shed) before a byte is read, and `Admitting` still holds a drain until the admission is queued (D-179). Not for listeners outside a group: one socket shared by all workers would wake every worker for each connection. `tests/accept_on_worker.rs`: no `net-io-accept` thread exists, 64 connections served, a closed door holds connections until it opens (twice), a group added and removed at run time refuses connections once the removal returns, the control line drains a server whose only listener is a group, and a dropped server's group refuses connections Preliminary result (tls-net, 2026-11-08, 6f7d1a2, load rising from 19 to 53, wide spreads, tokio's own run disturbed): CPU per handshake about 6% under D-194 (395 against 419 µs with the request after Finished, 450 against 480 µs with it; tokio-rustls 371 and 375 µs); `poll` 0 samples per 10,000 handshakes (27 before); `epoll_wait` 34 and `accept4` 37 against tokio's 11 and 26; `shutdown` 57 against 18, the largest system-call gap left (D-198). Clean numbers to follow from a quiet machine |
| D-198 | 2026-11-08 | A graceful close on the reactor calls `shutdown(SHUT_WR)` only when the socket lingers (input may still arrive: mid-exchange, readable, or the session spoke while idle, D-192). When it does not linger, the `close` that follows at once sends the FIN behind whatever is still queued, so the `shutdown` changed nothing on the wire | tls-net's profile of D-194 counted `shutdown` at 39 samples per 10,000 handshakes against tokio-rustls's 15: a closing reply after a whole request needs no linger, and its connection made one system call more than it needed. Saves one system call per such close [est. about 1 µs on Linux; tls-net measures]. Security and behaviour: none change. Linux and Windows both close gracefully (no `SO_LINGER` 0) with the queued bytes, then FIN; input that arrives after the close is answered with a reset with or without an earlier `shutdown` (the socket is closed for reading either way), which is why the linger exists for the cases that need it; abortive closes and soft closes are unchanged. The blocking driver keeps its `shutdown`: its linger decides after it, with one zero-timeout poll |
| D-199 | 2026-11-08 | A refusal is counted in `Stats::refused` and `refused_by` before its socket is shut or dropped, not after | `tests/server.rs::test_per_source_cap_per_listener` failed once in a full Windows run (8 reruns passed): the client saw the reset, the test read `refused_for(SourceCap)` as 0, and the accept thread then counted it. A program that reads `stats` after a client saw a refusal would see the same. Cost: none (the same two relaxed adds, earlier). Security: none; `Event::Refused` still goes out after the close |
| D-200 | 2026-11-08 | On the reactor a TLS connection's read gives back the session's input and output (`release_idle`) as a plaintext connection's read does, in any phase, before it looks at the TLS buffers (D-181); shipped as 0.7.1 | tls-net measured 42,030 bytes per warm idle TLS connection on 0.7.0 against 8,971 cold (tokio-rustls 11,651), with its own session holding 96 bytes more: the TLS branch of `read` released `cin` and `cout` but never called `release_idle`, so the output stayed attached with the spare buffers of the last reply (two 16 KiB after a 32 KiB page, up to four), charged to the shared budget. `test_idle_tls_connections_after_a_large_reply_hold_no_buffers` fails on 0.7.0 with a 2 MiB budget (100 warm idle connections, `OutputFull`) and passes with the fix. Cost: none on the data path (a check of the input and output when a TLS connection waits; the next exchange takes an output from the worker's spares, as a plaintext one does). Security: an idle TLS connection no longer holds budget, so warm idle connections cannot spend it for others; the wipe on release is unchanged. A patch, not 0.8: tls-net moves to 0.7 now, and the change is a single call on one path. Result (tls-net, `perf/idle.sh` on 0.7.1, 9,000 connections, WSL at load 100 to 120): 5,884 bytes per warm idle TLS connection against tokio-rustls's 11,680 in the same run (0.50x; 42,030 on 0.7.0), and 5,738 cold (tokio-rustls 11,418 to 11,447 in earlier runs); all 9,000 open on each stack. That meets tls-net's idle-memory target |
| D-201 | 2026-11-11 | `SPOKE_IDLE` (D-192) is set only for bytes sent while idle with no input heard in the exchange: a `HEARD` bit in the link's flag byte is set when the session is given input (`on_data` on both drivers, and a read that returns bytes) and cleared when the exchange settles back to `Idle` with nothing left over. On the pool, a kill that is net-io's own (the drain's deadline, a stop: `DRAIN_KILL` with its cause claimed) and lands during a graceful close's linger ends the linger and keeps the close's reason, as the reactor's `close_all` already did; a user's kill during the linger is still `Killed` and a reset | PR #12's CI failed `close_reasons::reason_drained` on every Linux run of the pool driver (`--no-default-features` and the forced-blocking job): `Killed(KillReason(0))` instead of `Drained`. Bisected to 52d625a (D-192). A session that sets `Idle` before it writes the reply (the usual way to end an exchange in `on_data`) had every reply counted as spoken while idle, so every close after an answered exchange lingered, which D-192 said it would not; the drain's deadline then killed the pool's linger, `Link::finish` reported the raw `Killed(0)` without the cause mapping `end` does, and `count_drained` counted it graceful. The local Linux gates had run `--all-features` only, where the reactor is the driver; they now run default, all, and no default features, as CI does. Windows CI passed `reason_drained`, where the linger's 2 s bound and the drain's 2 s deadline race, but the new test shows the same linger there. `tests/close_linger.rs` fails on b79502e on the Windows pool (the drain waited out a 4 s linger; the deadline-ended linger closed as `Killed(0)`) and passes with the fix on both drivers. Cost: none in memory (a spare bit), one OR per `on_data`, one AND per settle, one test per send. Security: no weaker than 0.7 for an answered connection (the shortcut it had); a server-first connection keeps D-192's linger, and the linger stays bounded in time and bytes. A drain report now counts such a connection graceful, which it was: its reply and FIN had gone out |
| D-202 | 2026-11-11 | A flush then close (a denial whose bytes went out, `close_after_flush`) lingers in any phase on both drivers: the pool's `linger_gracefully` skips the idle shortcut for it, and the reactor's `close` passes it to `begin_graceful_close` as input that may come. A graceful close keeps the shortcut (D-192, D-201) | Register R-47: on the Linux reactor at load 110 a soft kill reached a connection before its first `on_data`, so its phase was `Idle` and nothing was readable, and the close took the shortcut meant for an idle connection with nothing to protect. A flush then close is the server ending the connection on its own terms, whatever the peer is doing: its request may be crossing the close, seen or not, and a close at once answers it with a reset that can destroy the bytes just sent, the loss D-182 exists to prevent. `test_a_close_after_flush_lingers_before_any_input` failed every run on b79502e on both Windows drivers and passes with the fix. Cost: a soft-closed connection that never sends holds its descriptor and its soft-close place until the peer's FIN or the bound (`denial_linger`, 100 ms by default; the bound given to `close_after_flush`), where before an idle one closed at once [est. only under a burst of denials at accept, and capped by `denial_linger_max`]. No cost on the data path. Security: the linger stays bounded in time, bytes, and places, and a kill still upgrades it to a reset |
| D-203 | 2026-11-11 | 0.7.2 and 0.6.3: every overlapped call `sys/pipe.rs` makes sets the low bit of its event, so the kernel queues no completion packet for it; and a worker's completion port keeps the address of each operation it started until that operation's packet is dequeued, so a packet it did not start is counted and skipped, never read as an operation or freed | On the reactor a named pipe's handle is on its worker's port, and a denial goes out through the pipe stream's own overlapped write (`send_denial_once`): that write's packet reached the port, which took a pointer into the stream's own memory, already freed by the close, for one of its operations. A debug build panicked the worker; a release build corrupted its heap (an independent review, 2026-11-11, reproduced both). The low bit removes the cause; the address set makes the port safe against any other overlapped call on one of its handles. Cost: an insert and a remove in a hash set per operation on the worker's own thread [est. 20 to 40 ns each against a system call of 1 to 2 us], and about 10 bytes per operation in flight, with room for 1,024 made at the start so the steady state allocates nothing [est. 18 KiB per worker up front; 0.5 MB for 50,000 idle connections with a zero-byte receive each]; none for the event bit. Security impact: closes a memory-safety defect on Windows pipes under the reactor. `tests/pipe.rs::test_a_denial_to_a_pipe_client_that_does_not_read` (ended the process with `STATUS_HEAP_CORRUPTION` before) and `sys::iocp::tests::test_a_foreign_packet_is_skipped` (an access violation before) |
| D-204 | 2026-11-11 | 0.7.2 and 0.6.3: a Windows hand-off takes no socket that is not inheritable: the sender's duplicate is (its flag), while every socket net-io and std make is not and adoption clears the flag, so a description that names a socket the process already holds is refused and that socket is left as it was, neither wrapped nor closed. And `receive_shared` checks the sender before it reads: the far end of the pipe must run as this process's user (its token's user SID) and must have started no later than a second after the pipe connected, besides matching `expect.pid` when set | A description names its socket by the handle value it has in the receiving process, and `WSASocketW` returns whatever socket that value names, so a forged frame could make the receiver adopt one of its own sockets (two owners), or close it when verification then failed; the adopted list only knew sockets from earlier hand-offs. And the only sender check was `expect.pid`, while a receiver that dials a pipe another user created gets that user's frame. Why the inherit flag: the receiver cannot learn where a handle value came from, and this is the one mark a fresh duplicate carries that the receiver's own sockets do not; an inheritable socket made some other way is still caught by the user check, which bounds a forger to processes of the same user (who can already open this process). Why the start time: a sender that exited and whose number went to another process of the user before the check would pass the user check. Why no new handshake step (a nonce the receiver picks): the frame format is what an old process sends a new one during a restart across versions, and a patch must not break that. Cost: per hand-off one `OpenProcess`, two token reads, one `GetProcessTimes`, and one `GetHandleInformation` per socket [est. under 100 us once per restart]. Security impact: a hand-off can no longer give one socket two owners or close a socket it did not make; a sender of another user is refused. Known and unchanged: a fresh duplicate is inheritable from the moment the sender describes it until `receive_shared` adopts it, so a process this one starts in that window with handle inheritance gets a copy (a test that spawned a child while another test held one showed it); Windows offers no way to make the duplicate without the flag. `handoff::tests::test_a_description_naming_a_socket_owned_here_is_refused` and `test_a_forged_hand_off_neither_adopts_nor_closes_a_listener_here` failed before; `test_a_forged_hand_off_leaves_other_handles_alone`, `test_only_a_sender_of_this_user_that_was_already_running_is_accepted` |
| D-205 | 2026-11-11 | 0.7.2 and 0.6.3: a pipe name holds no `\` or `/`, is not `.` or `..`, and has at most 247 UTF-16 units (the OS takes 256 for the whole path, `\.\pipe\` included); otherwise `InvalidConfig` on both `PipeListener::create_with` and `PipeConnection::connect` | Windows rewrites a `\.\` path before opening it (`/` to `\`, `.` and `..` resolved), so `../C:/file` opened a file for reading and writing and `../UNC/host/share` a network path. A name with no separator is one component and cannot leave `\.\pipe\`. Why refuse `/` rather than open names literally with `\?\pipe\`: a name with `/` used to mean a nested pipe name (`a\b`), so either choice changes such a name; a refusal says so at once instead of silently reaching another pipe. Cost: one scan of the name per create or connect. Security impact: a pipe name taken from configuration or a peer reaches only a pipe. `pipe::tests::test_names_are_checked_before_any_call` failed before |
| D-206 | 2026-11-11 | 0.7.2 and 0.6.3: on the reactor a TLS connection with no bytes read ahead waits for its first bytes with a zero-byte receive and takes its two buffers only when they arrive (after a PROXY header it still takes them at start); each worker caps the bytes its unfinished handshakes hold at `buffer_budget / 4 / workers` (counted in bytes, so a listener with a larger read class weighs more; at least one handshake), and a handshake above it closes with `Limit(BufferBudget)`; worker pools keep at most `buffer_budget / workers` in their free lists and free the rest back to the budget, and a take whose class has nothing cached and finds the budget short frees the pool's cached buffers of other classes first (largest first, none when they could not cover it); outputs set aside while idle give their spares to the pool, and `Output::refill` makes room the same way; freed buffers are wiped first; `tls_released` is an explicit flag and `tls_restore` refuses to replace buffers in use | Silent TLS connections each held 128 KiB of the shared budget from accept to the handshake deadline, so about budget / 128 KiB of them (4,096 at the default) stalled every connection that needed a fresh buffer on any listener, and a pool never gave budget back, so after such a flood other size classes stayed starved until restart (an independent review, 2026-11-11). Found while fixing it: a flush or a graceful close while `cin` was out on a receive took a second pair of buffers and dropped live ones without refunding the budget. Cost: 0 bytes of budget per silent TLS connection, 131,072 before (measured); one zero-byte wait per handshake [est. 1 to 5 us]; a flood of stalled half-hellos above the cap closes new TLS handshakes on that worker until those time out, while plaintext keeps three quarters of the budget; refunds only off the steady path, plus one relaxed atomic read in `cx.output()` when the output has no free buffer and the pool caches other classes. Security impact: the budget bounds active handshakes and comes back after any burst; wipe-on-release covers freed buffers. `tls::test_silent_tls_clients_do_not_stall_plaintext_connections`, `tls::test_handshakes_holding_buffers_are_capped` and `reactor::test_buffers_one_class_freed_serve_another` failed on 0.7.1 |
| D-207 | 2026-11-11 | 0.7.2 and 0.6.3: a Unix listener never sets mode or owner through a path another user can change. The socket's directory is opened once and checked from its metadata. Writable by others without the sticky bit: refused with `InvalidConfig { field: "path", reason: InsecureDirectory }` (a new `ConfigReason` variant). Owned by this user or root and writable by nobody else, or sticky: bind in place, then `fchmodat` and `fchownat` (no link following) by name on the directory descriptor, `listen`, and `renameat` for `bind_atomic`. Otherwise: bind as `s` in a new `.nio<8 hex>` directory (0700, opened without following a link and checked to be a directory this user owns that nobody else can enter), set mode and owner there, `listen`, move it into place with `renameat` (`bind_atomic`) or `linkat` (`bind`, so a taken name still fails with `StaleSocketInUse`), then remove the private directory relative to its descriptors. Supersedes D-091's lstat, chmod, lstat sequence | `chmod` by path follows a link, and D-091's identity check ran only after the change, so anyone able to rename entries in the socket's directory could point the (often privileged) binder's mode or owner change at any file (an independent review, 2026-11-11, won the race in 228 ms). Why refuse a world-writable directory without the sticky bit: anyone there can also replace the finished socket, which no bind can prevent. Cost [est.]: once per bind and nothing per connection: a directory open, an `fstat`, and one or two `*at` calls in place; a `mkdir`, an open, a link or rename and two unlinks more, about 5 to 20 us, through a private directory. Ops: such directories must be made sticky or private; a crash mid-bind can leave an empty `.nio*` directory; through a private directory the socket's directory path plus 15 bytes must fit the path limit, and `getsockname` reports the private name, as `bind_atomic` already did; the directory must be readable by the binding user (a 0300 directory now fails with `PermissionDenied`). Security impact: closes the link redirect of mode and owner. Left: through a private directory, a swap of its entry before the bind can make the binder create a dead socket named `s` in another directory, with nothing applied to it (Linux could close it by binding through `/proc/self/fd`, FreeBSD with `bindat`; recorded for 0.8). `local::test_bind_refuses_a_directory_others_can_rename_in` and `local::test_socket_swapped_for_a_link_never_changes_its_target` failed on 0.7.1 |
| D-208 | 2026-11-11 | The independent review of 0.7.1 (2026-11-11: two critical, three high, six medium, eleven low, eight notes) is answered in three releases: 0.7.2 and 0.6.3 carry the critical and high fixes (D-203 to D-207), cut from their tags (0.6.3 on a `release/0.6` branch, with CI enabled for it) and published before anything is yanked; then 0.6.0 to 0.6.2, 0.7.0, 0.7.1, and 0.4.0 (which has the D-207 defect and no line to patch) are yanked; 0.8.0 merges 0.7.2 and fixes the mediums and the rest it can. The public changelog and release notes describe each defect plainly, with no steps to reproduce | A patch is where a security fix belongs, and every published dependent resolves to one without a manifest change (serve-net 0.7.1 and bouncer-io 0.9.3 ask for `^0.6.2`, tls-net 0.9.1 for `>=0.7, <0.8`, smtp-net 0.7.1 for `^0.7.1`); yanking after the patches keeps every dependent buildable. The mediums change behaviour more than a patch should or need the 0.8 reactor, and 0.8 is days away. Cost: two releases and one merge; none at run time beyond D-203 to D-207 |
| D-209 | 2026-11-11 | 0.8: on the reactor a named pipe's stream is closed dropping any write of its own still pending (`finish_reactor` marks a stream that does not half-close abortive), whatever the close reason | Since D-182 a denial close is no longer abortive, so a denial written to a pipe client that does not read left the stream's own write pending at close, and dropping the stream waited up to a second for it on the worker's thread, stalling every connection of that worker. Every other byte a reactor pipe sends goes through the driver and is done before the close. Cost: none; a pipe client that does not read loses a denial it was not reading, as a TCP client behind a reset would. Security impact: one silent client per denial can no longer stall a worker for a second |
| D-211 | 2026-11-12 | A reactor worker whose loop ends on a panic past its inner guards or a failed wait is marked dead under its inbox lock: hand-overs (`least_loaded`, a listener's feed, the inbox push) skip it, what waited in its inbox closes with a reset (tickets and caps returned, each close counted), its per-worker listening socket (D-197) passes to the next live worker, and `Stats::workers_lost` counts it; with every worker dead, new connections are refused as `Busy`. Marked dead, not restarted | Nothing marked a dead worker, so about one connection in `workers` went to an inbox nobody read, kept its caps and descriptor, and held `drain`, `shutdown` and `Drop for Server` for ever (an independent review, 2026-11-11: 4 of 8 clients unanswered, a 500 ms shutdown not back after 10 s). Why not a restart: the waker is a `OnceLock` read on every hand-off and would have to become replaceable, a wait that keeps failing would restart in a loop, and a worker that panicked past its guards has state that cannot be trusted. Cost: one relaxed load per worker looked at on a hand-off [est. 1 ns]; none otherwise. Security impact: no connection is silently swallowed; a server with no live worker refuses instead of hanging. `tests/shim.rs::test_a_reactor_worker_that_dies_takes_no_more_connections` failed before |
| D-212 | 2026-11-12 | A `Link`'s kill slot is held by a `SlotLease` (`sys/kill.rs`) that drops before the stream and, when no close path ran (an unwinding panic), runs `begin_close` then `uninstall` for its own generation only | A link dropped by unwinding closed its socket but left its slot active with the old generation and descriptor number, so a `KillHandle` used later passed every check and shut down whatever then held that number, against T-17. The lease carries the generation, so a stale lease never ends another connection's slot. Cost: 8 bytes per connection (the Windows reactor connection 904 to 912 bytes); no latency. Security impact: T-17 holds on unwinding too. `tests` `test_kill_after_an_unwound_link_is_refused` failed before; loom model `loom_unwound_lease_keeps_a_stale_kill_off_a_reused_number` |
| D-213 | 2026-11-12 | A drain returns at most `DRAIN_GRACE` (2 s) after its kill; a stop then waits up to 2 s for its worker threads (100 ms when the drain already gave up) and leaves the rest detached, counted in `Stats::workers_detached`; a listener removal's drain has the same bound. Accept threads are still joined without a bound | On the blocking driver a kill shuts the socket down, but a handler that computes, sleeps, or waits on a lock never returns, and `drain`, `shutdown` and `Drop for Server` waited for it, past the deadline the README promises. Cost: a stuck stop returns about deadline plus 2.1 s; a normal drain is unchanged. Security impact: one handler can no longer hold a restart for ever; a detached thread keeps its socket until its handler returns (documented). Left: a gate whose `admit` never returns still holds a stop, since accept threads are joined (R-51) |
| D-214 | 2026-11-12 | `build` and `add_listener` validate limits against the resolved driver's defaults (`Limits::validate_for`); the public `Limits::validate` keeps the blocking defaults | With `buffer_budget` unset, a reactor server with more than 512 workers was refused (`BudgetBelowWorkers` from the 64 MiB blocking default) though the reactor default is 512 MiB. Cost: none |
| D-215 | 2026-11-12 | `NoGate` implements `Gate` (admits everything; the write-path check is skipped), so a builder without `.gate(..)` builds | `ServerBuilder::new().listener(..).sessions(f).build()` did not compile until the caller found `.gate(AllowAll)`, against LIBRARY-SPEC-STANDARD 2.10. Why not make `AllowAll` the default type: code that names `NoGate` in a type would stop compiling. Cost: none |
| D-216 | 2026-11-12 | On Linux `sendfile` raises no SIGPIPE: SIGPIPE is blocked on the calling thread for the call, and a SIGPIPE the call raised (it failed with EPIPE) is consumed with a zero-timeout `sigtimedwait` before the mask is restored; one already pending before the call is left alone | `sendfile` takes no flags, so `MSG_NOSIGNAL` cannot apply, and a peer that reset mid-file killed a process whose SIGPIPE disposition is the default (a C host, a CLI that restores it), against T-32. Why per call: caching the thread's mask is racy, blocking it for a whole transfer would run gate code with SIGPIPE blocked, and `splice` plus `send` costs more calls. Cost: two `rt_sigprocmask` per `sendfile`, 250 to 420 ns measured under load against 11 to 24 us for a 64 KiB `sendfile` (2 to 4%) [est. less on a quiet host]. macOS and FreeBSD already set `SO_NOSIGPIPE` on sockets net-io makes; a socket a caller hands to the public `Driver` there needs it set by the caller (documented). Security impact: a reset peer cannot end the process. `test` with a child process at `SIG_DFL` died of signal 13 before |
| D-217 | 2026-11-12 | `Operation::SendFile { len: 0 }` completes as `Ok(0)` in `submit` with no system call on every driver; the Unix `send_file` returns `Ok(0)` for 0 and the IOCP one refuses 0 with `InvalidInput` as backstops | 0 means "to the end of the file" to `TransmitFile` and to macOS and FreeBSD `sendfile`, so an empty range computed by a `Driver` user sent the rest of the file (Linux sends nothing). Cost: none. Security impact: an empty range sends nothing anywhere. `tests/driver.rs` completed `Ok(4096)` on Windows before |
| D-218 | 2026-11-12 | `recv_file`'s splice stage reports socket-side and file-side errors apart; a file-side failure is fatal (`NetError::Os { op: "write" }`, and the connection closes with its class) and never falls back to `recv`; after a file refuses `splice`, later receives copy from the start, as D-170 says | A fallback `write` failing with EINVAL or EOPNOTSUPP after bytes had left the socket for the pipe was taken for splice being refused: the pipe's bytes were lost and later bytes landed at the wrong offset. Cost: none on the steady path. Security impact: no silent data loss. A new shim site `Op::FileWrite`; `tests/shim.rs` returned `Ok(196608)` with 64 KiB missing before |
| D-219 | 2026-11-12 | A blocking Unix connect interrupted by a signal waits for writability within the remaining `SO_SNDTIMEO`, reads `SO_ERROR`, checks for a peer, and connects again with the time left if the kernel dropped the attempt; a non-blocking connect's EINTR counts as in progress | A signal during `connect` reported a spurious failure. Cost: only on EINTR. `tests` returned `Interrupted` before |
| D-220 | 2026-11-12 | A systemd take that fails part way closes every descriptor of its plan, also those not yet reached | The ones not yet claimed stayed open with no owner for the life of the process. Cost: none |
| D-221 | 2026-11-12 | UDP: budgets are kept per source prefix (`UdpOptions::source_prefix(PrefixPolicy)`, default `PrefixPolicy::DEFAULT`: /32 IPv4, /64 IPv6) in one bounded table, and the amplification allowance per exact address in a second; a new or re-created source starts with `new_source_burst` (default 16, zero refused), not a full burst; `UdpBudget::GLOBAL_DEFAULT` (100,000 datagrams/s, 150 MB/s, burst 10,000) is the socket's default global budget | The table keyed the full address, so one host with a /64 got a fresh full budget per address (1,000 addresses, 3,000 datagrams through at a burst of 3) and evicted real sources; and with no global budget by default, 300,000 spoofed sources at one instant got 296,004 datagrams through, against API.md's "protects itself by default" and T-36. Cost: one more hash lookup per admitted datagram [est. 15 to 25 ns]; about 200 bytes per source, 12.5 MiB with the table full (about 130 and 8.5 MiB before) [est.]; a server above 100,000 datagrams/s must raise the global budget; a brand-new source sending more than 16 datagrams at once loses the rest until its bucket refills. Security impact: per-/64 budgets match TCP's caps, and a spoofed flood reaches the session at the global rate at most (10,000 of the 300,000 now) |
| D-222 | 2026-11-12 | UDP receive: a non-blocking receive returns "nothing ready" after 64 dropped datagrams or skipped reports in one call; host and network unreachable, and on Unix EHOSTDOWN, ENONET, ENOPROTOOPT, EPROTO and EMSGSIZE, and on Windows WSAENETRESET, are counted and skipped like a refused port; `SIO_UDP_NETRESET` is off on every Windows UDP socket; `recv_batch` off Linux puts the socket back in the mode it found it in | While every queued datagram was over budget a non-blocking receive never returned (one call held 757 ms and dropped 149,247 datagrams on Windows loopback), so the server's turn bound and its stop and drain checks waited; other ICMP-driven errors ended a user's receive loop; and `recv_batch` left a non-blocking socket blocking. Each skipped error is one a receive gives for no other reason, so a lasting fault still surfaces. Cost: one extra wait per 64 drops under a flood [est. under 2%]. Security impact: the loop keeps reaching its checks during a flood |
| D-223 | 2026-11-12 | A PROXY v2 LOCAL header is accepted whatever its family byte; an unknown family or a short block discards the address block, and a valid one still has its TLVs read; `V2BadFamily` and `V2AddressTooShort` apply to PROXY commands only | The specification (section 2.2) says a receiver must accept LOCAL and discard the address block; family 0x11 with no address was refused. Cost: none. Security impact: none; LOCAL uses the real endpoints |
| D-224 | 2026-11-12 | Windows hand-off: the claim on a received socket is its inherit flag, read and cleared under one lock; the list of adopted handle values (D-204's second check) is gone | A value on the list stayed taken for the life of the process, so after its owner closed it and Windows reused the value, a genuine hand-off naming it was refused and its fresh socket leaked. The flag belongs to the handle and ends with it. Cost: one uncontended lock per described socket. Security impact: unchanged against forged descriptions (D-204); a socket whose owner makes it inheritable again falls back to the same-user check, like any inheritable socket |
| D-225 | 2026-11-12 | On Windows `ListenOptions::reuse_address(true)` is ignored: listeners stay exclusive (`SO_EXCLUSIVEADDRUSE`) | `SO_REUSEADDR` on Windows lets another local socket bind over a live listener (port hijack), which a portable program asking for Unix semantics does not mean; measured here, an exclusive listener rebinds at once in every close order, even with an accepted connection still open, so nothing the option gives on Unix is lost. Cost: none. Security impact: closes a hijack through portable code |
| D-226 | 2026-11-12 | A graceful pipe close waits for its last write from a per-thread credit of one second that refills at 100 ms per second; abortive closes never wait | A client that never reads stalled the closing thread for a second per connection (three closes, 3.02 s). Cost: under attack, a close past the credit drops its unsent bytes, as a reset would. Security impact: a client cannot hold a thread a second at a time |
| D-227 | 2026-11-12 | Every extern declaration in `src/sys/` is checked against `libc` or `windows-sys` by type (Unix: function pointer types; Windows: each argument's size, signedness or pointer kind), with the missing field offsets and constants and `sigset_t`'s layout; the kernel-shared state of a pipe stream (its `OVERLAPPED`s and buffers) lives in an allocation reached only through raw pointers; `dev/SYS-SURFACE.md` is corrected | The headers of `sys/unix.rs` and `sys/windows.rs` and SYS-SURFACE claimed every signature, layout and constant was checked, and about 20 Unix and 27 Windows signatures, several offsets and constants were not (all were right by hand); and the kernel wrote a pipe's `OVERLAPPED.Internal` and read buffer while Rust held `&mut` to them. Cost: none at run time. Checked by building for macOS (x86_64, aarch64), FreeBSD, musl and aarch64 Linux; run on Linux and Windows |
| D-228 | 2026-11-12 | The 2026-11-11 review's findings are all answered (R-48 closed): D-203 to D-207 in 0.7.2 and 0.6.3, D-211 to D-227 in 0.8.0; its four notes on documented behaviour (pipe squatting with one instance, address keying edges, PROXY facts a consumer validates, stale status lines) are written into docs/API.md and docs/SECURITY.md and the status lines | Every finding has a fix with a regression test that failed before, or a documented behaviour with the reason. Cost: as each row states |
| D-229 | 2026-11-12 | Release text (README, CHANGELOG, release notes, crate docs and comments, PR and commit messages) names no decision or register ids and reads as the fix itself: what was wrong, what it does now, who should upgrade. The ids stay in this log and in `dev/ROADMAP.md`; `docs/API.md` and the design docs keep their "decision D-nnn" links into this log, which is where a reader goes for the reason | a changelog is read by people upgrading, who need the change and not the bookkeeping behind it |
| D-230 | 2026-11-18 | 0.8.0 is released on its soak at e40a63f (the commits after it change text and formatting only): 24 hours on each driver on Linux, 24 hours on each driver on Windows one at a time, and 5.5 hours on each driver on macOS (GitHub run 43149642369), all clean. R-50 and R-52 close with no recurrence in the soaks or on CI; one Windows CI failure of the in-process restart test is entered as R-53 | the soak is the 0.8 release gate in the roadmap, and nothing it exercises changed after it began |
---
## 15. Spikes at the design lock
Throwaway programs outside the crate, run on the reference box (Windows 11 and WSL2 Ubuntu) on 2026-10-23. macOS and FreeBSD are not available locally; where a spike asks about them, the 0.3 (macOS) and 0.5 (FreeBSD) CI integration tests decide, and the design has a fallback for each.
Windows 11 build 10.0.26200.9711; WSL2 kernel 6.6.87.2 (Ubuntu 24.04.4). Each observation was repeated at least three times with the same result.
| Spike | Question | Result | Decision |
|---|---|---|---|
| S-1 | Can a process that receives a listening socket by `WSADuplicateSocketW` associate it with its own IOCP port? | Only if no process associated it before. If the sender had associated it, the receiver's `CreateIoCompletionPort` fails with error 87, also after the sender closes its socket and port, and the receiver's `AcceptEx` completions arrive on the **sender's** port with the sender's key. A listener never associated works normally in the receiver (association and `AcceptEx`). `NtSetInformationFile(FileReplaceCompletionInformation)` moves the association but also redirects the sender's I/O. A `WSAPROTOCOL_INFOW` used twice returns the same handle; used after the sender closed, a dead handle (later calls fail with 10038). | D-016, D-054 |
| S-2 | Windows: is a socket usable after an `SO_RCVTIMEO` timeout, and what wakes a blocked `recv`? | After the timeout (10060) the socket stayed usable and 256 KiB sent afterwards arrived intact. `CancelIoEx` from another thread wakes a blocked `recv` at once with 10004 (`WSAEINTR`) and leaves the socket usable. `shutdown(SD_RECEIVE)` and `shutdown(SD_BOTH)` do not wake it. | D-052 (timeouts not relied on); `CancelIoEx` kept for the IOCP reactor |
| S-3 | Linux: does `shutdown` wake a blocked `recv` or `accept`? | `shutdown(SHUT_RD)` and `SHUT_RDWR` wake a blocked `recv`, which returns 0 (indistinguishable from end of stream without the closing flag). On a listener, `shutdown(SHUT_RD)` wakes `accept` with `EINVAL` and disables the listener for good (new connects get `ECONNREFUSED`). | D-052, D-057 |
| S-4 | Which options set on a listener are inherited by accepted sockets? | Linux and Windows plain `accept`: all tested options inherited (`TCP_NODELAY`, `SO_KEEPALIVE`, keepalive idle, `SO_RCVBUF`, `SO_SNDBUF`, `SO_LINGER`, `IP_TTL`). Windows `AcceptEx`: TCP-level options early, `SO_RCVBUF`, `SO_SNDBUF`, `SO_LINGER` only after `SO_UPDATE_ACCEPT_CONTEXT`. | D-055 |
| S-5 | Windows: does `WSAPoll` report a refused non-blocking connect? | Yes, every time (`POLLWRNORM \| POLLERR \| POLLHUP`, `SO_ERROR` 10061), after about 2.03 to 2.07 s (Windows retries the SYN). `select` reports it in the exception set only. `SIO_TCP_INITIAL_RTO` with no SYN retransmissions cuts the refusal to under 0.5 ms. | D-056 |
| S-6 | Linux: is renaming a bound Unix socket into place atomic? | Yes on ext4 and tmpfs: connects to the final name reach the same listener, the temporary name gives `ENOENT`, and about 14 million connects racing 4,500 renames over a stale socket file never saw `ENOENT`. `getsockname` keeps reporting the temporary name. | D-058 |
macOS and FreeBSD: S-3 (`shutdown` waking `recv`), S-4 (inheritance), and S-6 (rename) are checked by integration tests in CI at 0.3 (macOS) and 0.5 (FreeBSD); the design does not depend on the answer (the kill also writes the wake pipe; options are applied explicitly there; `bind_atomic` falls back to `bind` then `chmod` with the private-directory requirement if a test shows `rename` is not atomic).
---
## Appendix A: edge cases
Every item has a test by the version that ships the feature it concerns.
**Reads.** 1. `recv` returns 0 in Idle: `on_peer_closed`, default `PeerClosed`. 2. `recv` returns 0 in Receiving (truncated request): `on_peer_closed`; the session decides. 3. Partial reads at every split of a request (property test, every byte boundary). 4. The buffer is full and the session consumed nothing: `Input::is_full`; consume or grow, else `Limit(ReadBuffer)`, never a spin. 5. `EINTR`: retried with the remaining deadline. 6. Spurious `EAGAIN` after a readiness event: re-arm. 7. `ECONNRESET`: `PeerReset`. 8. `ETIMEDOUT` from keepalive probes: `Io(TimedOut)`, distinct from our deadlines. 9. Data after our `shutdown(Read)`: never delivered. 10. Data while parked stays in the kernel; read after wake. 11. Buffer budget exhausted: reading pauses and resumes; no error to the peer. 12. Zero-length buffer to `recv`: `Ok(0)` without a system call, documented as not end of stream. 13. 107 bytes without CRLF: `V1TooLong`. 14. v2 signature split across 12 segments: waits, then succeeds. 15. PROXY header and application data in one segment: `consumed` leaves the rest. 16. A second PROXY header is application data. 17. First byte of a PROXY header, then silence: handshake deadline.
**Writes.** 18. Partial writes continue from the exact offset (shim: every split). 19. `send` returns 0 for a non-empty buffer: treated as `WouldBlock` once, an error the second time. 20. `EAGAIN`: wait for writability within the write deadline. 21. `EPIPE`: `BrokenPipe`, no signal. 22. `ECONNRESET` mid-write: `PeerReset`; output wiped and released. 23. Write deadline with output pending: `WriteTimeout`, abortive. 24. Vectored write accepted partly into the third slice: slices advanced in place. 25. More than 64 chunks: several system calls, each after a gate check. 26. Gate denial between two chunks of one response: no denial, RST. 27. Gate denial before the first byte: denial written once (at most 512 bytes), then RST. 28. Denial while the denial is being written: not checked again; the write budget bounds it. 29. Output budget full: `OutputFull`, `Step::Write`. 30. `sendfile` short, `EAGAIN`, or the file shrinks: stop, `Io(FileTruncated)`. 31. TLS ciphertext larger than the remaining budget: flushed in pieces under the same budget.
**Close.** 32. Peer FIN with our output pending: finish writing, then close. 33. Peer RST while we write: `PeerReset`. 34. Graceful close with unread input: linger up to 2 s / 64 KiB, then close. 35. `SO_LINGER {1, 0}` produces RST on every OS (packet check where possible). 36. Kill during graceful linger: abortive at once. 37. Peer half-closes and keeps reading a streamed response: allowed. 38. `shutdown(Write)` twice: no-op. 39. Closing after the peer closed: no error.
**Accept.** 40. `EMFILE` with the reserve available: one accepted and reset, backoff starts. 41. Reserve cannot be reopened: backoff continues, reserve retried each step. 42. `ENFILE`: same routine. 43. `ECONNABORTED`: counted, continue. 44. `EPERM` (Linux firewall): counted, continue. 45. `ENOBUFS` / `ENOMEM`: pause. 46. `EBADF` / `EINVAL`: listener failed, event, no spin. 47. Peer address unspecified after accept: close quietly, count. 48. 10,000 connections in the backlog when the door opens: batches interleaved with IO. 49. Door closed with 500 in the backlog: they stay for the next process or are reset by `shutdown`. 50. Accept on a handed-off listener: effective backlog reported as unknown where it cannot be read. 51. IPv4-mapped peer on a dual-stack listener: normalized before caps and the gate. 52. IPv6 scope id: kept; the prefix key ignores it.
**Timers.** 53. Deadline exactly at the current tick: fires this tick. 54. Deadline in the past: next tick. 55. Beyond 64^6 ticks: overflow list, fires correctly across the wrap. 56. 100,000 timers in one slot: at most 1,024 per iteration, IO interleaved. 57. Deadline extended a million times: one entry, fires once. 58. Cancelled after firing but before processing: ignored (generation). 59. Clock jumps an hour: every expired timer once, in order, bounded per iteration. 60. Frozen clock: nothing fires, no spin. 61. Tick 100 ms with a 50 ms deadline: rejected at build.
**IOCP.** 62. `CancelIoEx` returns `ERROR_NOT_FOUND`: success. 63. Operation succeeds after the cancel: processed, bytes discarded, close proceeds. 64. Completion for a reused slot: generation mismatch, buffer returned, dropped. 65. Error packet with a null `OVERLAPPED`: driver shutdown path. 66. A client resets between the accept thread's `accept` and the worker's port association: the association or first receive fails, counted, closed quietly. 67. An accepted socket is associated with exactly one worker's port, never the listener (asserted). 68. Zero-byte receive completes and the peer had closed: the following `recv` returns 0. 69. Skip-completion-port mode on immediate success: processed inline; the flag-off path tested by forcing it off. 70. A socket associated with another worker's port: impossible by construction; tested.
**io_uring (P3).** 71. `io_uring_setup` `EPERM`: epoll fallback, event. 72. CQ overflow impossible by sizing; checked each loop. 73. Multishot accept ends: re-armed. 74. Zero-copy send: buffer released after the notification CQE. 75. Cancel of a completed operation: exactly one completion.
**Unix sockets and descriptor passing.** 76. Path exactly at the maximum: accepted; one more byte: `PathTooLong`. 77. NUL in the path: rejected. 78. Stale socket file: removed, bind succeeds. 79. Live listener on the path: `StaleSocketInUse`, file untouched. 80. Regular file, directory, symlink: not removed, error. 81. `MSG_CTRUNC`: every descriptor closed, `Truncated`. 82. Descriptors with a truncated or bad-checksum body: all closed. 83. More descriptors than declared: all closed, `CountMismatch`. 84. Peer credentials unavailable: `None`, and `require_uid` refuses. 85. Our path replaced by someone else's socket: `Drop` does not unlink it.
**Named pipes.** 86. All instances busy: `WaitNamedPipeW` within the timeout. 87. Next instance created before hand-over: never "no instance". 88. Client gone before `ConnectNamedPipe` completes: `ERROR_NO_DATA` handled; `ERROR_PIPE_CONNECTED` is success. 89. Backslash in the name part: rejected before the system call.
**UDP.** 90. Datagram larger than the buffer: `truncated`. 91. Zero-length datagram: valid, not end of stream. 92. ICMP port unreachable (`ECONNREFUSED`, Windows `WSAECONNRESET`): never fatal to the socket (Windows `SIO_UDP_CONNRESET` checked at 0.4). 93. Source table full: strangers share the overflow budget; verified peers keep entries. 94. Factor 1 and a 1-byte request: a 2-byte reply refused until `mark_verified`.
**Hand-off and inheritance.** 95. `LISTEN_FDS=0`: empty list. 96. `LISTEN_FDS` non-numeric, negative, or above 64: `BadCount` / `TooMany`. 97. No `LISTEN_PID`: empty list. 98. Names count differs: `NamesMismatch`. 99. Descriptor 3 closed or a regular file: `NotASocket`. 100. A connected socket: `NotListening`. 101. FastCGI descriptor 0 is `/dev/null`, a TTY, a pipe, a connected socket: `None`. 102. Receiver timeout: `Timeout`, nothing adopted. 103. Sender dies mid-send: nothing adopted. 104. Windows blob used twice in the target process: refused by the nonce check before `WSASocketW` (Windows itself would hand back the same socket). 105. A listener received on Windows is never associated with a completion port (asserted; D-016).
**Server lifecycle.** 106. `open_door` twice: no-op. 107. `close_door` before `open_door`: no-op. 108. `drain` with nothing open: returns at once. 109. `drain` deadline already passed: kills at once. 110. `shutdown` takes `self`, so a session cannot call it; `ServerHandle::request_stop` from a session thread is safe. 111. `adopt` while draining: `ShuttingDown`. 112. `release_listeners` with the door open: duplicates returned while this process keeps accepting (D-042); `shutdown` afterwards does not close them. 113. `kill` with an id from another server or a closed connection: `false`. 114. Control line `Normal` -> `Shed` -> `Normal` in one iteration: only the final state, idempotent. 115. `Draining` without a deadline: the drain timeout. 116. More workers than CPUs: allowed; placement wraps. 117. `post` after shutdown: `ServerStopped`. 118. Session panics in `on_data` (unwind): `SessionPanicked`, worker continues, buffers returned, one closed event.
**Configuration.** 119. Zero caps, workers, backlog, batch: `InvalidConfig`. 120. Per-listener above global: `InvalidConfig`. 121. Caps plus reserves above `RLIMIT_NOFILE`: `InvalidConfig` with have and need. 122. `ProxyMode::Required` without trusted proxies: `InvalidConfig`. 123. 1,025 trusted blocks: `TooMany`. 124. `Cidr::parse("10.0.0.1/8")`: `HostBitsSet`; `new_masked` accepts. 125. `/33`: `PrefixTooLong`. 126. Address in use: `AddrInUse` with advice. 127. Port below 1024 without privileges: `PermissionDenied` with advice. 128. IPv6 address with IPv6 disabled: `AddrNotAvailable`.
**Cross-thread.** 129. Kill from 64 threads at once: one close, one event, 63 `false`. 130. Kill racing `Step::Close`: one close. 131. Post racing close: delivered before close or counted stale, never delivered elsewhere. 132. `stats()` while workers run: consistent per-worker snapshots, no torn counter.