ktstr 0.23.0

Test harness for Linux process schedulers
# taobench port fidelity validation

This file is committed evidence that ktstr's native taobench port
(`WorkType::Taobench` / the `ktstr-taobench-validate` driver) reproduces the
ACCESS PATTERN of the reference taobench — a GET-dominated key-value object cache
with a fast in-cache hit path and a slow backing-store-miss path, driven to a
steady-state hit ratio.

Unlike the schbench validation (a per-flag-axis numeric envelope vs gcc/clang
builds of the same source), this is a **structural** comparison. The reference
taobench's published throughput is a many-core, many-instance figure (its
documented hit-ratio band is ~0.88–0.90), not head-to-head with a single host-side
engine run or an in-VM ktstr guest. The port is faithful when it reproduces the **shape** — GET-dominated
access, self-healing SET-on-miss, a fast≫slow split, a steady-state hit ratio
engineered by key-range-vs-capacity, and small-heavy long-tail value sizes — and
computes the same headline metrics (`total_qps = fast + slow`,
`hit_ratio = fast/(fast+slow)`) by the same formulas. Magnitude (absolute qps) is
environment-bound and is reported as context, not as a pass/fail target.

## Method

The same access pattern is realized two ways and compared for shape — the ktstr
port measured host-side (the numbers in ## Measured), the reference taobench against
its documented behavior (its harness's reported metrics + the documented 0.88–0.90
hit-ratio band; a single-host server + client build exercises the same access pattern):

| implementation | how |
|---|---|
| reference taobench | built from the reference's upstream source (its cache server + load-generator client); server + client run on disjoint cores over loopback with TLS disabled |
| ktstr port | `ktstr-taobench-validate` (the native engine via `run_standalone`), run host-side |

The reference server prints `fast_qps` / `hit_rate` / `slow_qps` every
stats-interval; the parser derives `total_qps = fast + slow` and
`hit_ratio = fast/total`. The port's driver prints the same fields computed by the
same formulas. Both are sized for a ~0.9 target hit ratio.

## Per-aspect comparison

Each row: the aspect, the reference taobench behavior, the port's behavior, and the
verdict (match, approximate, or intentional divergence).

| aspect | reference taobench | ktstr port | verdict |
|---|---|---|---|
| request mix | GET-dominated; SET only as the miss-fill the server tells the client to write (self-heal) — the GET-only warmup fills the cache via the same self-heal path | GET-only lookups; a miss inserts (fills) the key on the slow path — the same self-heal | match |
| key distribution | ~uniform random over a key id range (not zipfian); hit ratio set by RANGE size vs cache capacity | uniform random over `[0, key_range)`; `key_range = capacity_objects / target_hit` | match |
| hit ratio | ~0.88–0.90, an emergent property of a CAPACITY-BOUNDED LRU cache whose working set exceeds the cache's EFFECTIVE object capacity (eviction ↔ refill equilibrium; the reference tunes its key range with a correction factor so the steady state lands on the target) | settles to `target_hit_pct` via a bounded FIFO-evicting sharded cache with `key_range > capacity` (eviction ↔ refill equilibrium); pinned by the in-process test (`0.80..=0.97`, not drifting to 1.0) | match (mechanism; FIFO ≈ LRU for uniform access) |
| value sizes | small-object-heavy size profile (the reference's documented average item size is ≈ 434 B); the server clips every served item to a 64 KiB maximum, so served bytes cap at 64 KiB | representative small-heavy long-tail distribution (mean ≈ 332 B, tail to 64 KiB); bytes are allocated per the distribution and TOUCHED on serve (the cache memory-bandwidth cost) | approximate (own distribution, same small-object-heavy shape + 64 KiB ceiling + same touch) |
| fast / slow tiers | fast worker thread serves a hit (`fast_qps`); a miss is enqueued to a slow dispatcher pool that simulates a backing-store fetch (sleep) then SET-fills (`slow_qps`) | a client thread serves its own hit inline (fast path); a miss is handed to a slow dispatcher thread that sleeps the backing-store fetch then fills + wakes the client | match (slow tier present; client+fast merged — see divergences) |
| slow-path service time | a FIXED per-request `usleep(tao_slow_path_sleep_us)` — no jitter, no tail. (The `[target/2, 2·target]` uniform jitter + idle-poll backoff of "smart" mode is the reference's dispatcher/slow-thread/worker POLL & idle-wait sleep, a separate knob, not the per-request fetch.) | fixed `slow_path_sleep_us` by default; `slow_path_p99_us > slow_path_sleep_us` switches to a per-fetch heavy-tailed Pareto with median `slow_path_sleep_us` and 99th percentile `slow_path_p99_us` (clamped at a 2s max) | enhancement (the port adds a true heavy-tailed per-request service-time distribution; the reference's per-request service time is fixed) |
| headline metrics | `fast_qps`, `slow_qps`, `total_qps = fast+slow`, `hit_ratio = fast/total`; per-interval `hit_rate = 1 - get_misses/get_cmds` | same five, same formulas (`write_taobench_scalars`); request-time `get_cmds`/`get_misses` vs response-time `fast_ops`/`slow_ops` split preserved | match |
| arrival model | closed-loop, fixed-connection-count load generator (fixed `--clients` / `--threads`, no rate target; each connection issues its next request on the prior's completion); its harness reports throughput + hit ratio, not request-latency percentiles | closed-loop by default (`arrival_rate == 0`); `arrival_rate > 0` switches to OPEN-loop fixed-rate arrival and measures serve latency from the intended arrival (coordinated-omission) | match (both closed-loop); port adds an open-loop CO mode (enhancement) |
| serve latency | the reference's harness surfaces throughput + hit ratio only, no request-latency percentiles | open-loop coordinated-omission serve latency, per-phase `taobench_serve_{p50,p90,p99,p999,min,max}_us` + whole-run `taobench_serve_*_us_whole`, measured from intended arrival via schbench's `PlatStats` histogram | enhancement (the port adds an open-loop CO serve-latency the reference harness does not report) |

## Measured

Both sized for a ~0.9 target hit ratio. The comparison is **shape** (steady-state
hit ratio + fast≫slow composition), not absolute magnitude.

**Reference taobench.** The reference's headline result is a many-core,
multi-process aggregate: its harness scales to multiple server instances across the
host's cores and reports per-interval `fast_qps` / `slow_qps` / `total_qps` and a
`hit_rate`, with a documented steady-state hit-ratio band of **0.88–0.90**. It does
not surface request-latency percentiles, and its absolute qps is environment-bound
(a many-core, multi-process aggregate) — so magnitude is not a head-to-head target.
Only the shape is compared: a steady-state hit ratio in the 0.88–0.90 band reached
by eviction↔refill, a fast≫slow composition, and the same `total_qps` / `hit_ratio`
formulas.

**ktstr port.** `ktstr-taobench-validate` host-side — a single in-process engine.
Reproduce with:

```
cargo run --release --bin ktstr-taobench-validate \
  --features cli-bins,integration -- -m 128 -c 16 -H 90 -r 15
```

(128 MiB cache, `target_hit_pct 90`, 16 client + 5 slow threads, closed loop; one
representative run, 14.91 s measured window):

| metric | value |
|---|---|
| hit_ratio (fast/total) | **0.8999** |
| hit_rate (1 − misses/cmds) | 0.8999 |
| fast_qps | ~292,100 |
| slow_qps | ~32,500 |
| total_qps (fast + slow) | ~324,600 |
| fast : slow split | ~9 : 1 |

The port settles to **0.8999** — its 0.90 target — by the same eviction↔refill
equilibrium that holds the reference inside its 0.88–0.90 band. (hit_ratio ==
hit_rate here because a non-phased whole-run run drains all in-flight requests, so
response-time and command-time counts coincide; the two diverge only across a
phase boundary, the intended non-tautology.) Absolute qps is a single in-process
host run on one machine — environment-bound, not comparable to the reference's
many-core aggregate, and not a pass/fail axis.

**Shape match:** the port settles to 0.8999 (its 0.90 target) with a ~9:1
fast:slow split (fast:slow = hit:miss at the target ratio), inside the reference's
documented 0.88–0.90 band. Both reach the steady-state hit ratio as an emergent
eviction↔refill property and compute `total_qps` / `hit_ratio` by the same
formulas — the structural fidelity claim. Absolute qps differs by environment
(many-core multi-process reference vs a single in-process engine) and is not a
comparison axis.

## Documented divergences

The port models the access pattern and the CPU / memory / scheduling
characteristic, not the wire protocol. Deliberate divergences from the reference:

- **No sockets / TLS.** The reference is a cache server + a load-generator
  client over a real loopback socket; the port is entirely in-process (client →
  fast → slow via condvar/queue handoffs). The scheduler-relevant load is the
  thread wakeups + the slow-path off-CPU + the cache lock contention, which the
  in-process model preserves; the socket I/O layer is dropped (as is TLS).
- **Client + fast tier merged.** The reference has separate client connection
  threads and server fast worker threads; the port's client thread serves its own
  hit inline. The slow dispatcher tier is kept separate (the miss handoff + wakeup
  is preserved).
- **No OOM path.** The reference reports `slow_qps_oom` and OOM lines under memory
  pressure; the port's bounded-evicting cache models steady state only, so
  `slow_qps_oom` is always 0 (not ported).
- **FIFO vs LRU eviction.** The reference evicts LRU; the port evicts FIFO. For a
  uniform key stream the equilibrium hit ratio is `cap/key_range` under either
  policy, so the steady-state hit ratio matches.
- **Value-size distribution is an approximation,** not a copy of the reference's
  value-size table — the port uses its own small-heavy long-tail histogram (mean
  ≈ 332 B, 64 KiB ceiling) approximating the reference's small-object-heavy profile
  (documented average ≈ 434 B, served items clipped at 64 KiB), so no external data
  table is carried.
- **Serve-path latency is coordinated-omission-corrected, open-loop only.** Under
  open-loop arrival (`arrival_rate > 0`) the port measures serve latency from the
  INTENDED arrival time (so a backlog from slow service inflates the late requests'
  latency instead of being omitted) into schbench's `PlatStats` histogram, exposed
  as the per-phase `taobench_serve_*_us` and whole-run `taobench_serve_*_us_whole`
  percentiles. It is CO-corrected (measured from intended arrival, not issue time)
  and recorded only in open loop — the default closed loop has no intended-arrival
  schedule, so the serve-latency keys read absent there. The reference harness
  surfaces no request-latency percentile, so this is a port-only metric, not a
  head-to-head latency axis.
- **Heavy-tailed slow-path service time (port enhancement).** The reference's
  per-request slow-path service time is a FIXED `usleep(tao_slow_path_sleep_us)` —
  no jitter, no tail. (Its `[target/2, 2×target]` per-thread-cached uniform jitter +
  idle-poll exponential backoff, "smart" mode's `my_nanosleep`, governs the
  reference's dispatcher/slow-thread/worker POLL & idle-wait sleeps — a separate
  knob, not the per-request fetch.) The port adds a true per-fetch heavy-tailed
  service time: with `slow_path_p99_us > slow_path_sleep_us`, each miss's
  backing-store fetch is a Pareto draw (median `slow_path_sleep_us`, 99th percentile
  `slow_path_p99_us`, clamped at a 2 s maximum), so the open-loop serve-latency tail
  reflects realistic backing-store variance the reference's fixed service time
  cannot. `slow_path_p99_us == 0` (default), `<= slow_path_sleep_us`, or
  `slow_path_sleep_us == 0` keeps the fixed-latency legacy behavior.
- **Per-phase metric surface.** The port's qps/hit_ratio are `MetricKind::PerPhase`
  (surfaced via the per-phase `PhaseBucket` / `VmResult::phase_metric` path, like
  schbench's latency metrics), not the run-level cross-run scalar fold.
- **Magnitude is not comparable.** The reference's numbers are a many-core,
  multi-process run; the port is a single in-process engine. Only the shape
  (composition + hit ratio) is compared.