# vmrunner
## Motivation
`vmrunner` exists to run real payloads locally in lightweight VMs, without requiring
cloud infrastructure such as GCP, AWS, Azure, ..
Particularly useful for software that needs to interact with an operating
system boundary, filesystems, block devices, networking, namespaces, or root-only
operations which ones does not grant access to on the host.
## Goals
- Spawn VMs with the least privileges possible.
- Use existing tools where complexity mandates:
- use `mkosi` for building images
- use `libkrun`/`libkrunfw` for VM boot/runtime
- use existing distro VM qcow2 images
- Allow for ensamble tests using an overlay network
- Work with minimal host setup on Linux and macOS _hosts_
- Support Linux _guests_ only
- Focus on integration tests rather than long-running VM orchestration.
- Enable ephemeral invasive tests
- filesystem-level operations
- mount and remount behavior
- block image manipulation
- root-like guest operations
- backup/restore tooling that needs realistic filesystem state
## How
`vmrunner` composes existing lower-level projects and adds a small Rust-facing testing
layer on top. The core model is:
1. select or build a Linux _guest_ image
2. ensure a Linux _guest_ init sidecar exists
3. create a per-test writable qcow2 overlay
4. configure a `libkrun` VM
5. direct-exec the requested guest command
6. collect stdout/stderr/status back into the Rust test
### Crates
#### `vmrunner`
The top-level crate provides the user-facing test integration:
- `#[vmrunner::test(...)]` macro support,
- root image caching,
- guest init sidecar management,
- platform child-process setup for Linux/macOS hosts,
- re-exports of the VM test harness API.
#### `vmrunner-test-harness`
The runtime harness launches `libkrun` VMs and wires them into the Rust test process.
It provides:
- `TestCase`, `Node`, and `Network` APIs,
- one forked host process per VM,
- guest stdio capture,
- per-node writable qcow2 overlays,
- virtio-net backed by Unix streams,
- optional host uplink backends for Linux/macOS,
- direct execution of a guest command.
#### `vmrunner-macros`
The macro crate turns a Rust test into a VM-aware integration test wrapper.
It handles:
- default Linux guest target selection,
- Linux-only guest target validation,
- host-target maps for Linux/macOS hosts,
- setup object generation for the test body.
#### `vmrunner-sysroot`
The sysroot helper extracts compiler sysroots and arbitrary guest paths from VM disk
images via libguestfs. For Linux guests, `vmrunner` can use this to build the guest
`/init.krun` sidecar against headers/libraries extracted from the same root image.
Key properties:
- uses libguestfs rather than mounting guest filesystems with the host kernel,
- opens images read-only,
- supports explicit image formats such as `qcow2` or `raw`,
- supports manual mounts when libguestfs inspection is insufficient,
- keeps sysroot extraction separate from VM execution.
#### `mkosi`
`mkosi` is a good fit for producing controlled Linux OS images for tests. It can build
repeatable distro images with known packages, users, system configuration, and files.
`vmrunner` should not need to become an image builder. Instead, `mkosi` can own image
construction while `vmrunner` owns test-time execution.
Typical responsibilities:
```text
mkosi:
- compose Linux image
- install packages needed by the payload
- install development headers if a sysroot is needed
- produce a boot/root image artifact
vmrunner:
- cache/select image
- create writable overlay
- run test payload inside VM
- collect test result
```
#### `libkrun`
`libkrun` is the VM runtime used by the harness. It provides the low-level ability to
create a microVM, attach disks and network devices, configure the guest command, and
enter the VM.
`vmrunner` uses `libkrun` rather than implementing a VMM. This keeps the project small
and focused on test orchestration.
Relevant features used by `vmrunner` include:
- VM context creation,
- CPU/memory configuration,
- qcow2/root disk attachment,
- virtio console for stdio,
- virtio-net via Unix streams,
- overlay file injection,
- direct guest command execution.
#### `libkrunfw`
`libkrunfw` is loaded by the host-side `libkrun` process and provides the bundled Linux
kernel/firmware payload used by direct-exec mode.
Important boundary:
```text
libkrunfw.so is loaded on the host.
The Linux kernel payload from it runs in the guest VM.
```
`vmrunner` relies on this path for Linux guest direct-exec. That is why the guest side
is Linux-only.
#### VM images
Distro VM images provide realistic guest userspace and filesystem state. `vmrunner`
can use cloud images or locally produced images.
Examples:
- Fedora Cloud images,
- Ubuntu cloud images,
- mkosi-produced Linux images,
- local qcow2 images prepared by project-specific tooling.
`vmrunner` caches HTTP(S) images locally and creates per-test writable overlays so the
base image remains reusable.
#### qcow2 format / qcow2 tooling
`qcow2` is the primary image format used for root disks and overlays. `vmrunner` uses
existing qcow2 format support rather than inventing its own disk image format.
In the current implementation, qcow2 handling is used for:
- cached base images,
- writable per-node overlays,
- GPT inspection for selecting root partitions in distro images,
- non-destructive test isolation.
Conceptually:
```text
base image qcow2 read-only, shared between tests
|
v
per-test overlay writable, disposable
|
v
libkrun disk attached as guest root device
```
#### `unshare` (Linux host)
On Linux hosts, `vmrunner` uses `unshare(1)` for process-level namespace setup where
appropriate. This avoids performing namespace changes inside an already-running,
potentially multithreaded Rust test process.
The goal is least-privilege local testing:
- user namespace for root-like operations where possible,
- network namespace isolation for network tests,
- no cloud privileges,
- no long-lived privileged daemon owned by `vmrunner`.
#### binary signing (macOS host)
The current test binary must be ad-hoc signed with the Hypervisor entitlement and re-executed in a child process in order to be able to use `framework.Hypervisor`.
---
### Logic Flow
#### High-level test flow
```text
Rust test
|
| #[vmrunner::test(...)]
v
vmrunner-macros
|
| generates TestSetup
v
vmrunner crate
|
| ensure root image exists
| ensure Linux /init.krun sidecar exists
v
vmrunner-test-harness
|
| create per-node qcow2 overlay
| fork one host process per VM
v
libkrun host process
|
| load/use libkrunfw Linux kernel payload
| attach root disk overlay
| attach console/network devices
v
Linux guest VM
|
| /init.krun mounts/pivots to root image
| direct-exec requested command
v
Rust test receives stdout/stderr/status
```
#### Component relation diagram
```text
+----------------------+
| Rust tests |
+----------+-----------+
|
v
+----------------------+
| vmrunner-macros |
| target selection |
+----------+-----------+
|
v
+------------------+ +----------------------+ +----------------------+
| mkosi outputs | +----------+-----------+ +----------+-----------+
+------------------+ | |
| v
| +----------------------+
| | libkrun |
| | host-side VMM API |
| +----------+-----------+
| |
| v
+----------------------+ +----------------------+
| vmrunner-sysroot | | libkrunfw |
| libguestfs extract | | Linux kernel payload |
+----------------------+ +----------+-----------+
|
v
+----------------------+
| Linux guest VM |
| payload under test |
+----------------------+
```
#### Image and sysroot flow
```text
image source
+------------------------------+
| cloud image / mkosi artifact |
+---------------+--------------+
|
v
target/vmrunner-qcow2-imag
|
v
mkosi-prepared qcow
mandatory for known cloud images
install sysroot packages
|
+---------------+---------------+
| |
v v
vmrunner-sysroot qcow2 overlay creation
extract headers/libs per VM node
| |
v v
Linux compiler sysroot libkrun root disk
| |
v v
build /init.krun Linux guest boots/direct-execs
```
#### Host/guest boundary
```text
Linux/macOS host process
|
| dlopen/load host libraries
| configure libkrun VM
| attach disk/network/console
|
+--> libkrun
|
+--> libkrunfw loaded on host
| |
| +--> Linux kernel bytes made available to guest memory
|
+--> start vCPU
|
v
Linux guest
|
+--> /init.krun
+--> requested test payload
```
#### Least-privilege host flow
```text
Linux host test binary
|
| if namespace isolation is needed
v
unshare --user --map-root-user --net --fork
|
v
child test process
|
v
fork VM child process
|
v
libkrun_start_enter()
```
```text
macOS host test binary
|
| ensure Hypervisor entitlement on test binary
v
codesigned child process
|
v
fork VM child process
|
v
libkrun_start_enter()
```
### Motivation for `vmrunner-sysroot`
The `vmrunner-sysroot` extension exists because the code that prepares a VM often needs
to be built for the guest environment, not for the host environment.
For example, `vmrunner` injects a Linux guest `/init.krun` sidecar before executing the
payload under test. That init binary should be compiled against headers, libc, startup
objects, and linker behavior that match the guest image as closely as possible. Using
the host's `/usr/include` or host libc would make the build depend on the developer's
machine rather than the image being tested. While this often works, it will break
subtly every blue moon.
`vmrunner-sysroot` solves that by extracting headers and libraries from the VM image
itself, read-only, via `libguestfs`, yielding a sysroot derived
from the same distro image that will later run the payload.
- cross-compiling guest support code from Linux or macOS hosts,
- building guest init code against the guest distro's libc rather than the host libc
- keeping cloud images and `mkosi`-built images as the source of truth
- making integration tests more reproducible across developer machines and CI runners
The sysroot extraction is intentionally separate from VM execution. It is a build-time
or test-setup helper, while `libkrun` remains responsible for actually running the
Linux guest.