ktstr 0.23.0

Test harness for Linux process schedulers
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137

# ktstr

ktstr is a test harness for Linux process schedulers, with a focus on
sched_ext (BPF-extensible process scheduling). It boots Linux kernels
in KVM virtual machines with
controlled CPU topologies, runs workloads, and verifies scheduling
correctness. Also tests under the kernel's default EEVDF scheduler.

## Quick taste

The simplest test calls a canned scenario:

```rust,ignore
use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    scenarios::steady(ctx)
}
```

```sh
cargo ktstr test --kernel ../linux
```

Without a `scheduler` attribute, tests run under EEVDF. See
[Getting Started](getting-started.md) for testing a sched_ext scheduler.

## Library API

The `ktstr::prelude` module re-exports the types needed for writing
tests — see the full [API reference (rustdoc)](https://ktstr.dev/rustdoc/ktstr/)
for every workspace crate. Declare cgroups and workloads as data with
`CgroupDef`:

```rust,ignore
use ktstr::prelude::*;

#[ktstr_test(llcs = 1, cores = 2, threads = 1)]
fn my_test(ctx: &Ctx) -> Result<AssertResult> {
    execute_defs(ctx, vec![
        CgroupDef::named("cg_0").workers(2),
        CgroupDef::named("cg_1").workers(2),
    ])
}
```

The prelude also exports low-level types (`CgroupGroup`,
`WorkloadConfig`, `WorkloadHandle`) for manual cgroup and worker
management, `Assert` for composable assertion config, and
`WorkerReport` for telemetry access.

For binary workloads (running `schbench`, `fio`, or any external
executable as part of a test), see
[Payload Definitions](writing-tests/scheduler-definitions.md#derive-payload).
`#[ktstr_test(payload = FIXTURE)]` runs a `Payload` (binary
workload) alongside the cgroup workers; the `scheduler =` slot
takes a bare `Scheduler` reference — the const emitted by
`declare_scheduler!`.

## What it tests

- **Fair scheduling** -- workers get CPU time without starvation or
  excessive scheduling gaps.
- **Cpuset isolation** -- workers stay on assigned CPUs.
- **Dynamic operations** -- cgroups created, destroyed, and resized
  mid-run.
- **Affinity** -- scheduler respects thread affinity constraints.
- **Stress** -- many cgroups, many workers, rapid topology changes.
- **Stall detection** -- scheduler doesn't drop tasks.

## Design

Two principles drive ktstr's architecture:

**Fidelity without overhead** -- every test boots a real Linux kernel
in a KVM VM with real cgroups and real BPF programs. No mocking, no
containers, no shared state. The VMM is minimal: two 16550 serial
ports (COM1 for kernel console, COM2 for crash diagnostics) and a
virtio-console (virtio-MMIO on every arch) for the host↔guest TLV
stream on port-1 — used for scenario dispatch and result envelopes —
plus virtio-blk for file-backed block storage with optional btrfs
templates and virtio-net for in-VMM L2 loopback used by network
workload tests. On x86_64 virtio-blk and virtio-net are virtio-PCI
functions behind a host bridge with ECAM/CAM config-access windows
(auto-enabled by attaching a disk or NIC, or via `.pci(true)`); on
aarch64 they use virtio-MMIO.

**Direct access over tooling layers** -- the host-side monitor reads
guest memory directly via BTF (BPF Type Format)-resolved struct
offsets to observe scheduler state. The monitor runs entirely
host-side — no BPF programs are injected into the guest to collect
scheduler telemetry, so observations do not perturb scheduling
decisions. (BPF programs loaded by the scheduler under test, the
BPF verifier pipeline, and the auto-repro probe pipeline are
separate concerns; those are the code under test, not the
observation layer.) See [Monitor](architecture/monitor.md) for
details on BTF resolution and guest memory introspection.

## BPF verifier analysis

The `verifier_pipeline` tests boot a scheduler in a VM and capture
per-program verifier output from the real kernel verifier. The
default output applies **cycle collapse** to reduce repetitive loop
unrolling. See [BPF Verifier](running-tests/verifier.md) for details.

## Auto-repro probe pipeline

When a scheduler crashes, ktstr can automatically rerun the failing
scenario with BPF probes attached to the crash-path functions. See
[Auto-Repro](running-tests/auto-repro.md) for details.

## Workspace structure

| Component | Purpose |
|---|---|
| `ktstr` (lib) | Core library |
| `ktstr-macros` | `#[ktstr_test]`, `declare_scheduler!`, and `#[derive(Payload)]` proc macros |
| `ktstr` (bin) | Host-side CLI |
| `cargo-ktstr` (bin) | Cargo-integrated workflow: test, coverage, llvm-cov, kernel mgmt, verifier analysis, stats, interactive shell |
| `scx-ktstr` | Minimal BPF scheduler for testing |

`ktstr` and `cargo-ktstr` are the two user-facing `[[bin]]`
targets in the crate; install them with
`cargo install --locked ktstr`.
The crate also defines four test-fixture `[[bin]]` targets —
`ktstr-jemalloc-probe`, `ktstr-jemalloc-alloc-worker`,
`ktstr-schbench-validate`, and `ktstr-taobench-validate` —
used by the integration tests. They require the non-default
`integration` feature, so a default `cargo install` builds only the
two user-facing binaries and never places the fixtures on `$PATH`.

## Kernel config

`ktstr.kconfig` in the repo root contains the kernel config fragment
needed for scheduler testing (sched_ext, BPF, kprobes, cgroups).
Copy it to your kernel source tree and run `make olddefconfig`.