# go-lib v2 Roadmap — Addressing Known Limitations
This document is the step-by-step implementation plan for the eight known
limitations listed in `README.md`. Each step includes a scope description,
sub-tasks, and any inter-step prerequisites.
---
## Suggested release sequencing
```
v1.1 Steps 1, 2, 6, 7 — pure library / scheduler API, low risk
v1.2 Step 8 (Option A) — loom integration, CI hardening
v2.0 Steps 3, 4, 5 — assembly + kernel APIs, ship together
v2.1 Step 8 (Option B) — TSan CI pass post v2.0
```
Steps 3 → 4 → 5 form a natural dependency chain:
- Stack growth (Step 3) must land before async preemption (Step 4), because
the signal handler injects a preemption frame onto the live goroutine stack
and requires `g.stackguard0` to be valid.
- Async preemption (Step 4) should land before netpoll (Step 5) so that an
I/O-ready callback is never delayed by a CPU-bound goroutine hogging an M.
---
## Step 1 — `go_lib::sync::Cond`
**Effort:** small
**Release:** v1.1
Port Go's `sync.Cond` as a goroutine-aware condition variable. Using
`std::sync::Condvar` directly would block an OS thread (M), starving other
goroutines sharing that M.
### Sub-tasks
1. Add `src/sync/cond.rs`.
2. Define `Cond { waitq: Mutex<WaitQueue>, ... }` where `WaitQueue` is a
`VecDeque<*mut G>`.
3. Implement `wait(guard: MutexGuard<T>)`:
- Enqueue `current_g()` into `waitq`.
- Release `guard`.
- Call `gopark(WaitReason::CondWait)` to park the goroutine without blocking
the M.
- Re-acquire `guard` on return.
4. Implement `notify_one()`: pop one G from `waitq`, call `goready(gp)`.
5. Implement `notify_all()`: drain `waitq`, call `goready` on each.
6. Export as `go_lib::sync::Cond`.
7. Add unit tests (single-waiter, multi-waiter, spurious-wakeup resilience).
8. Add an example `examples/cond.rs` (producer / consumer with bounded buffer).
---
## Step 2 — Runtime-adjustable `GOMAXPROCS`
**Effort:** medium
**Release:** v1.1
Allow the number of Ps (and thus the degree of true parallelism) to be changed
at runtime, matching Go's `runtime.GOMAXPROCS(n)`.
### Sub-tasks
1. Add `GOMAXPROCS: AtomicUsize` field to `Sched`.
2. Implement `pub fn set_gomaxprocs(n: usize) -> usize`:
- Atomically swap the value; return the old value.
- If *increasing* by `delta`: allocate `delta` new `P` objects via `P::new`,
push each to the idle-P list, call `startm(ptr::null_mut())` for each new P.
- If *decreasing* by `delta`: mark `delta` surplus Ps `Pidle`; their M-threads
will call `stopm` on the next `findrunnable` cycle — no forced kill needed.
3. In `schedinit`, read the `GOMAXPROCS` environment variable (parse as `usize`,
clamp to `[1, 256]`) before falling back to `available_parallelism`.
4. Update `findrunnable` to consult the live `GOMAXPROCS` value when iterating
Ps for work-stealing so newly added Ps are visible immediately.
5. Tests:
- Verify P count before and after an increase.
- Verify P count before and after a decrease.
- Stress-test: spawn 64 goroutines while toggling GOMAXPROCS between 1 and 8.
---
## Step 3 — Goroutine stack growth (`morestack` / `copystack`)
**Effort:** large
**Release:** v2.0
**Prerequisite for:** Step 4
This is the most invasive change. Every goroutine currently gets a fixed 64 KiB
`mmap` region. Stack growth requires emitting stack-check prologues in the
context-switch assembly, a `morestack` trampoline, and a `copystack` routine
that relocates live stack frames.
### Sub-tasks
1. **Reduce initial stack size** — change the allocation in `stack.rs` from
64 KiB to 8 KiB. Update `g.stackguard0 = stack.lo + STACK_GUARD` where
`STACK_GUARD` is 256 bytes (matches Go's `_StackGuard`).
2. **Stack-check prologue** — in `asm_amd64.rs` (and `asm_arm64.rs`), add a
check at the top of `gogo` and at each `mcall` return:
```asm
; AMD64
cmp rsp, [rdi + offsetof(G, stackguard0)]
jbe morestack_trampoline
```
The offset of `stackguard0` must be a compile-time constant exported from
`g.rs` via a `const OFFSET_STACKGUARD0: usize`.
3. **`morestack` trampoline** (naked fn, Rust + inline asm):
- Save all caller-saved registers to the current goroutine stack.
- Switch to g0 via `mcall`.
- Call `newstack()` on g0's stack.
- Restore registers; return to the function that triggered the check.
4. **`newstack()`** (in `stack.rs`):
- Compute `new_size = old_size * 2`, capped at 1 GiB.
- `mmap` a new region with a guard page at the low end (`mprotect PROT_NONE`).
- Call `copystack(gp, new_stack)`:
- Walk the frame chain from `g.sched.sp` up to `g.stack.hi`.
- For each pointer-sized word in the old stack that falls within
`[old_lo, old_hi)`, adjust it by `delta = new_lo - old_lo`.
- Copy all bytes from `old_lo` to `old_hi` into the corresponding position
in the new stack.
- Update `g.stack`, `g.stackguard0`, `g.sched.sp`, `g.sched.bp`.
- `munmap` the old region.
5. **Frame-pointer requirement** — compile with `-C force-frame-pointers=yes`
(add to `.cargo/config.toml`) so `copystack` can reliably walk frames.
6. **Adjust `G::new`** — allocate 8 KiB, set `stackguard0` correctly.
7. **Remove the 64 KiB constant** from `stack.rs`; replace with `STACK_MIN =
8 * 1024` and `STACK_MAX = 1 * 1024 * 1024 * 1024`.
8. Tests:
- A goroutine that recurses 100 000 levels deep must not segfault.
- Verify `g.stack.hi - g.stack.lo` grows across recursive calls.
- Verify the old stack region is unmapped after growth (try reading from old
address; should SIGSEGV — use `catch_unwind` + `signal` in the test).
---
## Step 4 — Async (signal-based) preemption
**Effort:** large
**Release:** v2.0
**Prerequisite:** Step 3
Currently `sysmon` sets `g.preempt = true` but a goroutine is not actually
preempted until it voluntarily calls `gosched()` or blocks. Async preemption
injects a signal into the OS thread carrying the goroutine.
### Sub-tasks
1. **Signal handler registration** — in `schedinit`, install a `SIGURG` handler
via `libc::sigaction`. `SIGURG` is what Go uses; it is ignored by default
and not used by debuggers. The handler must be async-signal-safe.
2. **`asyncPreempt` assembly stub** (`asm_amd64.rs` / `asm_arm64.rs`):
- The signal frame already saves integer registers (the OS does this).
- Save FP / SIMD registers to the goroutine stack (the OS does not).
- Call `asyncPreempt2()` (a Rust function) via an indirect call so the return
address is on the goroutine's stack.
3. **`asyncPreempt2()`** (in `sched.rs`):
- Check `g.preempt`; if false, return immediately (spurious signal).
- Clear `g.preempt`.
- Call `mcall(preempt_fn)` → `schedule()`, yielding the goroutine.
4. **Signal delivery** — in `sysmon`, when a goroutine has run ≥ 10 ms:
```rust
libc::pthread_kill((*mp).pthread_id, libc::SIGURG);
```
Add `pthread_id: libc::pthread_t` to the `M` struct; set it in the M-thread
body via `libc::pthread_self()`.
5. **Guard against signal-in-g0** — check `current_g().is_null()` at the top
of the signal handler; if on g0 (or in `entersyscall`), return immediately.
6. Tests:
- A CPU-bound goroutine with no `gosched()` calls must be preempted within
~20 ms (measure with `Instant::now()`).
- A goroutine blocked in `gopark` must not receive spurious wakeups from
the signal.
---
## Step 5 — Netpoll / async I/O
**Effort:** large
**Release:** v2.0
**Prerequisite:** Step 4 recommended (not strictly required)
Add a `netpoll` subsystem so goroutines can wait on file-descriptor readiness
without blocking an OS thread. This removes the need for `with_syscall` on
network operations.
### Sub-tasks
1. **`src/runtime/netpoll.rs`**:
- `netpoll_init()` — create the `epoll` fd (Linux) or `kqueue` fd (macOS).
- `netpoll_arm(fd, mode, gp)` — register `fd` for read (`mode=0`) or write
(`mode=1`), storing `gp` in `epoll_data` / `kevent.udata`.
- `netpoll_unarm(fd)` — deregister.
- `netpoll_wait(timeout_ns: i64) -> Vec<*mut G>` — call `epoll_wait` /
`kevent` with the given timeout; return the list of ready Gs.
2. **Integrate into `findrunnable`**:
```rust
let ready = netpoll_wait(0); for gp in ready { goready(gp); }
```
3. **Integrate into `sysmon`** — when the run queues are empty, call
`netpoll_wait(sleep_ns)` so the process can sleep efficiently on I/O.
4. **`go_lib::net` facade** (`src/net.rs`):
- `TcpListener::bind(addr)` — wraps `std::net::TcpListener`; sets `O_NONBLOCK`.
- `TcpListener::accept() -> TcpStream` — calls `libc::accept4`; on
`EAGAIN`, calls `netpoll_arm(fd, READ, current_g())` then `gopark`.
- `TcpStream::read(&mut self, buf) -> usize` — same pattern for `EAGAIN`.
- `TcpStream::write(&self, buf) -> usize` — same pattern for `EAGAIN` on write.
5. **`with_syscall` remains** for file I/O (no `epoll` support for regular files
on Linux; use `io_uring` in a future step if needed).
6. Tests:
- Echo server + client goroutines exchange 10 000 messages without blocking
any OS thread beyond the M-threads.
- Verify that a goroutine waiting on `TcpStream::read` does not hold its P
(another goroutine on the same P must make progress).
---
## Step 6 — `panic` / `recover` across goroutine boundaries
**Effort:** medium
**Release:** v1.1
Currently a Rust panic inside a goroutine aborts the process. This step gives
each goroutine its own panic boundary and exposes a `recover()`-like API.
### Sub-tasks
1. **Wrap goroutine bodies in `catch_unwind`** — in `execute` (the entry point
called by `gogo`), wrap the goroutine closure:
```rust
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(f));
if let Err(payload) = result {
handle_goroutine_panic(payload);
}
```
2. **`handle_goroutine_panic`**:
- Store `payload` in a thread-local `GOROUTINE_PANIC: Cell<Option<Box<dyn Any>>>`.
- Call the user-set panic handler (see below), then clear the cell.
- Continue into `goexit0` / `schedule` so the scheduler keeps running.
3. **`go_lib::set_panic_handler(f: fn(&dyn Any))`** — registers a process-wide
handler called for every unrecovered goroutine panic. Default: print the
payload to stderr (mirrors Go's behaviour of printing the stack trace and
continuing).
4. **`go_lib::recover() -> Option<Box<dyn Any>>`** — if called from within the
panic handler or a deferred closure registered *before* the panic, drains
`GOROUTINE_PANIC` and returns `Some(payload)`; otherwise returns `None`.
This matches Go's `recover()` semantics closely enough for practical use.
5. Tests:
- A panicking goroutine does not abort the process.
- Other goroutines continue to run after a peer panics.
- `recover()` returns the correct payload.
- A panic in the main goroutine still propagates (no silent swallow).
---
## Step 7 — `go_lib::context`
**Effort:** small
**Release:** v1.1
A pure-library port of Go's `context` package. No scheduler changes required.
### Sub-tasks
1. Add `src/context.rs`.
2. Define `ContextInner`:
```rust
struct ContextInner {
deadline: Option<Instant>,
done_tx: Option<Sender<()>>,
done_rx: Receiver<()>,
parent: Option<Arc<ContextInner>>,
err: Mutex<Option<ContextError>>,
}
```
3. **`Context`** — a newtype `Arc<ContextInner>` with these methods:
- `done(&self) -> &Receiver<()>` — usable directly in `select!`.
- `deadline(&self) -> Option<Instant>`
- `err(&self) -> Option<ContextError>` — `None | Cancelled | DeadlineExceeded`.
4. **Constructors**:
- `context::background()` — root; never cancels.
- `context::with_cancel(parent: &Context) -> (Context, CancelFn)` — `CancelFn`
sends `()` on the done channel and sets `err = Cancelled`.
- `context::with_deadline(parent: &Context, t: Instant) -> (Context, CancelFn)` —
spawns a goroutine that calls `go_lib::sleep(t - now)` then cancels.
- `context::with_timeout(parent: &Context, d: Duration) -> (Context, CancelFn)` —
sugar over `with_deadline`.
5. **Parent propagation** — when a parent is cancelled, all children must also be
cancelled. Implement via a `children: Mutex<Vec<Weak<ContextInner>>>` list on
each `ContextInner`; `CancelFn` walks the list and cancels each child.
6. Export as `go_lib::context::{background, with_cancel, with_deadline, with_timeout, Context, CancelFn, ContextError}`.
7. Tests:
- Cancel propagates from parent to child.
- `done()` channel receives after cancel.
- Deadline fires automatically via `sleep`.
- `with_cancel` from a done-channel example in `select!`.
---
## Step 8 — Race detector
**Effort:** medium
**Release:** v1.2 (Option A) / v2.1 (Option B)
### Option A — `loom` integration (near-term, nightly not required)
1. Add `loom` as a dev-dependency under `[dev-dependencies]`.
2. Add a Cargo feature `loom` that replaces `std::sync::atomic`, `std::sync::Mutex`,
and `std::sync::Arc` with `loom::sync::atomic`, etc., via `cfg` re-exports
in a `src/loom_shim.rs` module.
3. The raw-pointer and assembly paths (`gopark`, `goready`, context switches)
cannot be modelled by `loom` — annotate them with `loom::skip` or keep them
outside the loom boundary.
4. Add a CI job:
```yaml
- run: cargo test --cfg loom
```
5. Run `loom` with `LOOM_MAX_PERMUTATIONS=10000` on the channel, WaitGroup, and
Cond tests.
### Option B — ThreadSanitizer (v2.0, requires nightly)
1. Add a CI matrix entry using the nightly toolchain:
```yaml
- run: cargo +nightly test -Zsanitizer=thread --target x86_64-unknown-linux-gnu
```
2. TSan instruments the assembly context switches and catches happens-before
violations across goroutine boundaries that `loom` cannot model.
3. Annotate known-benign intentional races (e.g., the `g.preempt` flag written
by sysmon without a lock) with `__tsan_acquire` / `__tsan_release` annotations
via `libc` bindings, matching how the Go runtime annotates its own races.
4. Fix any real races TSan surfaces before shipping v2.0.
---
## File map — new / changed files per step
| 1 | `src/sync/cond.rs`, `examples/cond.rs` | `src/sync/mod.rs`, `src/lib.rs` |
| 2 | — | `src/runtime/sched.rs`, `src/runtime/p.rs`, `src/lib.rs` |
| 3 | — | `src/runtime/stack.rs`, `src/runtime/asm_amd64.rs`, `src/runtime/asm_arm64.rs`, `src/runtime/g.rs` |
| 4 | — | `src/runtime/sched.rs`, `src/runtime/sysmon.rs`, `src/runtime/m.rs`, `src/runtime/asm_amd64.rs`, `src/runtime/asm_arm64.rs` |
| 5 | `src/runtime/netpoll.rs`, `src/net.rs` | `src/runtime/sched.rs`, `src/runtime/sysmon.rs`, `src/lib.rs` |
| 6 | — | `src/runtime/sched.rs`, `src/lib.rs` |
| 7 | `src/context.rs` | `src/lib.rs` |
| 8A | `src/loom_shim.rs` | `Cargo.toml`, `.github/workflows/ci.yml` |
| 8B | — | `.github/workflows/ci.yml` |