# PaperBoy
A Rust-native alternative to Postman. Collections are [Hurl](https://hurl.dev)
`.hurl` files and environments are `.vars` files (`KEY=value`), so everything is
plain text you can commit, diff and review. No hosted service, no telemetry,
nothing leaves your machine.
One binary, three front-ends over the same core:
| Terminal UI | `paperboy` | The default. Full client. |
| Graphical UI | `paperboy -g` | eframe/egui. Behind the `gui` Cargo feature. |
| Headless runner | `paperboy -c collection.hurl` | For scripts and CI. Also runs reports (`-r`). Exits non-zero on failure. |
Collections, environments, themes and the git workflows behave identically in
all three.
- [Install](#install)
- [Concepts](#concepts)
- [Terminal UI](#terminal-ui)
- [Graphical UI](#graphical-ui)
- [Environments and secrets](#environments-and-secrets)
- [Git remotes](#git-remotes)
- [Importing from Postman](#importing-from-postman)
- [Headless runner](#headless-runner)
- [Changelog](CHANGELOG.md)
## Install
### Prebuilt binaries
The quickest route, and the one that needs no Rust toolchain and none of the
[build prerequisites](#build-prerequisites) below. Download from the [latest
release](https://github.com/jhobern/paperboy/releases/latest), unpack, and put
`paperboy` somewhere on your `PATH`.
| `paperboy-<version>-linux-x86_64.tar.gz` | Linux x86-64 — glibc 2.35+ (Ubuntu 22.04+, Debian 12+, Fedora 36+) |
| `paperboy-<version>-macos-arm64.tar.gz` | macOS, Apple Silicon |
| `paperboy-<version>-windows-x86_64.zip` | Windows x86-64 |
| `paperboy-<version>-linux-x86_64-headless.tar.gz` | Linux x86-64, headless runner only |
Every file except `-headless` is the *same* binary in all three modes: it runs
`-c` headless, the terminal UI, and `--gui`. There is no separate GUI download,
and choosing the full build costs nothing if you never open it — the graphical
front-end loads its display libraries only when `--gui` is passed, so the binary
still runs on a machine that has none.
The Linux archives also need the system `libxml2` (`libxml2.so.2`), which
every desktop distribution installs and a minimal container image does not —
if that is where you are heading, use the image below. The Windows build links
libxml2 in and needs nothing; the macOS build uses the copy macOS itself ships
at `/usr/lib`, so it needs nothing either. RHEL 9 and Rocky/Alma 9 ship glibc
2.34 and so are *not* covered by the Linux archive; build from source or use
the container there.
Each release publishes `SHA256SUMS` beside the archives:
```sh
sha256sum -c SHA256SUMS --ignore-missing
```
### Container
For CI images and anywhere a slim base has no system libxml2 — which the Linux
binaries above need:
```sh
docker run --rm -v "$PWD:/work" ghcr.io/jhobern/paperboy -c collection.hurl
```
The image is the headless build, so it takes the same arguments and writes the
same reports as any other PaperBoy; `/work` is its working directory, which is
why the mount above is all the setup there is.
### From source
```sh
cargo install paperboy --locked # terminal UI + headless runner
cargo install paperboy --locked --features gui # …and the graphical UI
cargo install paperboy --locked --no-default-features # headless runner only
```
`--locked` is recommended: it builds the dependency versions PaperBoy was
tested against rather than re-resolving to whatever is newest. (A yanked
`arrayref` release took plain `cargo install` down on 2026-08-20 while
`--locked` kept working — and it took the terminal-only build with it, since
Cargo resolves optional dependencies whether or not their feature is on.)
The `gui` feature is opt-in because eframe/winit/wgpu roughly double the
dependency tree. Both builds share the same state file, so you lose nothing by
switching. Running `--gui` without it prints the command to install one with it.
`--no-default-features` turns the terminal UI off and leaves the headless
runner, which is the shape wanted for CI images and Docker containers: it drops
40 dependencies and about a third of PaperBoy's own source, none of which a
scripted `-c`/`-r` run would ever execute. The resulting binary takes the same
arguments and writes the same reports; only the interactive front-end is
missing, and running it with no arguments says so rather than doing nothing.
### Build prerequisites
Only when building from source — the [prebuilt
binaries](#prebuilt-binaries) and the container need none of this.
Five things Cargo can't fetch for you:
| macOS | `xcode-select --install` then `brew install pkg-config libxml2` |
| Debian/Ubuntu | `sudo apt install build-essential pkg-config libxml2-dev libclang-dev perl` |
| Fedora/RHEL | `sudo dnf install pkgconf-pkg-config gcc make perl libxml2-devel clang-devel` |
| Arch | `sudo pacman -S pkgconf base-devel perl libxml2 clang` |
| Alpine | `sudo apk add build-base pkgconfig perl libxml2-dev clang-dev` |
| Windows (MSVC) | see [Windows](#windows) below |
**libxml2 + `pkg-config`** because `hurl`/`hurl_core` depend unconditionally on
the `libxml` crate (Hurl's XPath asserts *are* libxml2's XPath engine), and that
crate is a binding to a system libxml2 rather than a vendored copy.
**libclang** because `bindgen` generates those bindings at build time. **A C
compiler, `perl` and `make`** because PaperBoy pulls `curl` in directly with
`static-curl`/`static-ssl`, so libcurl and OpenSSL are compiled from vendored
sources — which is why there is no `libcurl-dev` row above. The `gui` feature
adds no build-time requirement; its X11/Wayland libraries are `dlopen`ed at
runtime.
On Windows (MSVC) the same needs are met by different tools, so it gets its own
section below.
On macOS the Command Line Tools cover everything except `pkg-config`, which is
the failure most people hit. The second is subtler: macOS *does* ship libxml2,
but the SDK carries only its headers and a linker stub — there is no
`libxml-2.0.pc` anywhere, so `pkg-config` cannot discover the system copy and
the build fails even though the library is right there. The simple fix is
Homebrew's keg-only libxml2, pointed at explicitly:
```sh
brew install pkg-config libxml2
export PKG_CONFIG_PATH="$(brew --prefix libxml2)/lib/pkgconfig:$PKG_CONFIG_PATH"
```
That bakes an `/opt/homebrew/...` path into the binary, which is fine for a
local build and not for one you intend to hand to anyone else. The released
macOS archive instead writes a one-off `libxml-2.0.pc` pointing at the SDK, so
it links `/usr/lib/libxml2.2.dylib` and runs on any Mac; see the
`Point pkg-config at the macOS SDK's libxml2` step in
[`.github/workflows/release.yml`](.github/workflows/release.yml) if you need to
do the same. Note that static linking is *not* an alternative here: Homebrew's
bottle ships no `libxml2.a`, and even with one, Apple's linker has no
`-Bstatic`, so `-lxml2` against a directory holding both takes the `.dylib`.
#### Windows
Nothing here is unusual, but almost every piece arrives from a different place
than it does on Unix, so in order:
1. **Visual Studio Build Tools**, workload *Desktop development with C++* —
this is `cl.exe`, `nmake` and the Windows SDK. `cargo` finds them through
the registry, so they don't have to be on `PATH`; building from an **x64
Native Tools Command Prompt for VS** is still the least surprising way.
2. **libxml2, from vcpkg.** On MSVC the `libxml` crate asks vcpkg and nothing
else — there is no pkg-config fallback — and it finds vcpkg through
`VCPKG_ROOT` (or a previous `vcpkg integrate install`), *not* by looking for
`vcpkg.exe` on `PATH`:
```bat
git clone https://github.com/microsoft/vcpkg C:\vcpkg
C:\vcpkg\bootstrap-vcpkg.bat
setx VCPKG_ROOT C:\vcpkg
C:\vcpkg\vcpkg install libxml2:x64-windows-static-md
```
`vcpkg.exe` is called by its full path on purpose: bootstrapping builds it
inside the tree and adds nothing to `PATH`, so a bare `vcpkg install` in the
shell that just ran the bootstrap fails with "not recognized". In PowerShell
that is true even from inside `C:\vcpkg` — unlike `cmd`, PowerShell does not
search the current directory, so there it is `.\vcpkg install …`. If your
tree is somewhere with a space in the path, quote it (`"C:\Program
Files\vcpkg\vcpkg" install …`) and, in PowerShell, put `&` in front.
The triplet is the part worth reading twice. `x64-windows-static-md` is a
static libxml2 built against the *dynamic* CRT, which is what Rust's MSVC
targets link — it is what vcpkg-rs asks for by default, and installing the
plain `x64-windows` port instead is the usual first wrong turn. Use
`arm64-windows-static-md` on ARM64. (`VCPKGRS_DYNAMIC=1` switches to the DLL
ports, and `VCPKGRS_TRIPLET` overrides the choice outright — `build.rs`
honours both when it tells you which triplet is missing.)
3. **LLVM, Strawberry Perl and NASM:**
```bat
winget install --id LLVM.LLVM -e --accept-package-agreements --accept-source-agreements
winget install --id StrawberryPerl.StrawberryPerl -e --accept-package-agreements --accept-source-agreements
winget install --id NASM.NASM -e --accept-package-agreements --accept-source-agreements
```
One package per line, with the agreement prompts pre-answered, so the block
can be pasted in one go: a tool that stops to ask something reads the next
pasted line as the answer and quietly swallows a command. (`winget install
A B C` also works, on v1.5 and later.) Or `choco install llvm strawberryperl
nasm -y`, from an **Administrator** prompt. LLVM supplies `libclang.dll`
for bindgen — if it isn't found, point at it with
`setx LIBCLANG_PATH "C:\Program Files\LLVM\bin"`. Perl and NASM are for the
vendored OpenSSL build (`perl Configure`, then NASM for the crypto
assembly), and **NASM's installer does not add itself to `PATH`**, which
OpenSSL needs it to be — so add `C:\Program Files\NASM` yourself.
4. **A placeholder icon for `hurl`'s build script.** This one is an upstream
bug rather than a missing tool, and it stops the build outright:
```powershell
$cargoHome = if ($env:CARGO_HOME) { $env:CARGO_HOME } else { "$env:USERPROFILE\.cargo" }
$dir = Join-Path $cargoHome "registry\src\bin\windows"
New-Item -ItemType Directory -Force -Path $dir | Out-Null
Copy-Item paperboy.ico (Join-Path $dir "logo.ico") # any .ico will do
```
`hurl` 8.0.1 embeds a Windows icon from `../../bin/windows/logo.ico`, a path
that only resolves inside hurl's own git checkout; the published crate ships
nothing outside its own directory, so `rc.exe` reports `RC2135: file not
found` and the build script panics. It is not conditional, so **no Windows
build of anything depending on `hurl` 8.0.1 can succeed without this**.
`rc.exe` resolves the path from the build script's working directory, which
Cargo sets to the package root, so `../../` lands in `registry\src` — hence
the command above. The icon is linked into the finished executable, so use
one you are happy to see on `paperboy.exe`.
[Upstream has fixed it](https://github.com/Orange-OpenSource/hurl/issues/5207)
by skipping the icon when the file is absent, but the fix is unreleased and
8.0.1 is still the newest crate. PaperBoy's own `build.rs` cannot paper over
it: Cargo runs a dependency's build script before its dependent's, so hurl's
has already failed by the time ours would run. **The prebuilt Windows binary
is built this way and is unaffected** — this applies only to building from
source.
`build.rs` checks all of this before the build starts and names the triplet and
the tree it actually looked in, so a wrong `VCPKG_ROOT` reads as one line rather
than as a link error several minutes later.
PaperBoy's `build.rs` checks for all five before the build gets going and fails
with the package-manager command your machine actually wants (it detects
Homebrew/MacPorts, apt, dnf, yum, zypper, pacman, apk). It fails rather than
warns because Cargo runs build scripts concurrently and doesn't replay their
warnings — a warning lands dozens of `Compiling …` lines above the real error.
It never installs anything, and it can't prompt: a build script has no terminal.
Checks that could be wrong (cross-compilation, target-suffixed `PKG_CONFIG_*`,
the libclang heuristic) only warn, and `PAPERBOY_SKIP_DEP_CHECK=1` disables it
entirely.
If you have a libxml2 lying around and want to skip `pkg-config` and `bindgen`,
`libxml`'s build script takes an explicit path — `--config` reaches transitive
build scripts:
```sh
cargo install paperboy --locked \
--config 'env.LIBXML2="/opt/homebrew/opt/libxml2/lib/libxml2.dylib"'
```
Vendoring libxml2 instead isn't possible from here: the crate has no vendored
build, no `libxml2-src` exists, and it declares no `links` key, so there's no
`DEP_*` channel to reach into it with. It would have to be added upstream.
From a checkout:
```sh
cargo run # terminal UI
cargo run --features gui -- --gui # graphical UI
cargo run --no-default-features -- -c collection.hurl # headless only
cargo test # add --features gui for the GUI's tests
```
PaperBoy builds in four shapes — headless, terminal, terminal + GUI, and GUI
alone — and CI checks all four, because a configuration nothing builds is a
configuration that stops compiling. Each must also stay warning-free; the
dead-code analysis is carried by the two shapes that include the terminal UI
(see the note at the top of `src/main.rs` for why).
## Concepts
- **Collection** — a `.hurl` file: an ordered list of requests with method,
URL, headers, cookies, body/form fields and optional `[Captures]`/`[Asserts]`.
Postman `.json` exports open directly and are converted on the way in.
- **Environment** — a `.vars` file of `KEY=value` lines supplying `{{ VAR }}`
values. See [Environments and secrets](#environments-and-secrets).
- **Workspace** — a folder of collections, reports and environments, browsed
through a single tab as a filesystem tree.
- **Report** — a `.trail` file: a PaperTrail script that runs requests from a
collection, loops over environments or data, and writes CSV/JSON/HTML/XLSX.
Editable as text or as [blocks](#the-papertrail-block-editor); runnable from
the UI or [headlessly](#reports).
- **Generated value** — a `# [Gen]` row: an expression evaluated just before a
request is sent, supplying the nonces, timestamps and signatures a pre-request
script used to. See [Generated values](#generated-values).
- **Scratch Space** — tab 0. A collection with no file behind it until you save
it.
- **Request names encode folders.** `Auth/Tokens/Refresh` browses as a folder
path; Postman's folder structure imports into this automatically.
## Terminal UI
Press `?` or `F1` for the full, current key list. The essentials:
| `Tab` / `Shift+Tab` | Move focus between panes |
| `↑`/`↓`, `j`/`k` | Move selection |
| `←`/`→`, `h`/`l` | Switch tabs / scroll list text horizontally |
| `Enter` | Edit the selected request (or descend into a folder row) |
| `Shift+R` | Edit as raw Hurl text — for anything the form doesn't expose |
| `F5`, `Ctrl+Enter` | Run the current request |
| `Alt+F5` | Run the whole collection in one Hurl execution |
| `n` / `b` | New request / set the base URL |
| `f` / `s` | File menu / Settings menu |
| `Ctrl+S` | Save the open report, else the active collection |
| `[` / `]`, `PageUp`/`PageDown` | Previous / next tab |
| `Ctrl+Shift+←`/`→` | Reorder the active tab |
| `x` / `u` | Delete / undo — requests, tabs and environments each keep their own undo stack |
| `o` / `x` (report results) | Sort by the cursor's column / re-run the cursor's row — see [Reading a report's results](#reading-a-reports-results) |
| `/` | Find a request anywhere in the collection (whole tree on a Workspace tab) |
| `Alt+↑`/`↓` | Reorder requests — the order `Alt+F5` and the CLI follow |
| `m` / `c` | Move / copy a request to another collection in the workspace |
| `p` (Requests) | Link an environment to this collection |
| `a` (Response) | Build an assert or a capture from what came back |
| `a` (Env pane) | Make an environment active |
| `r` (Env pane) | Retry a failed secret lookup |
| `w` (Workspace tab) | Reopen the file-tree picker |
| `+`/`-`, `<`/`>` | Resize the response pane / left column |
| `q`, `Ctrl+C` | Quit |
In the request wizard:
| `[`/`]`, `PageUp`/`PageDown` | Switch section tab (`All│Headers│Cookies│Queries│Options│Form│Body│Asserts│Captures│Reports│Generated`). `[`/`]` only when focus isn't on a text field, so brackets stay typable |
| `Alt+1`–`9`, `Alt+0` | Jump straight to a section (`Alt` because most terminals can't report `Ctrl`+digit) |
| `Ctrl+↑`/`↓` | Previous / next section |
| `Ctrl+D` / `Ctrl+E` | Delete a row / toggle its enabled checkbox |
| `Ctrl+Z` / `Ctrl+Shift+Z` | Undo / redo within the focused text cell |
| `←` from a Key cell | Reach the enabled checkbox — it's the leftmost column |
| `Ctrl+F` or `Enter` on a File value | Open a file picker |
| `F2`, `Ctrl+Enter` | Save |
| `Esc` | Cancel (asks first if there are unsaved edits) |
Worth knowing:
- **`[Form]`/`[Multipart]`, `[Cookies]`, `[Captures]`, `[Asserts]` and
`[BasicAuth]`** are all editable as tables in the wizard; the expected status
is just an assert (`status == 200`). Saving picks the right Hurl section:
all-text fields become `[Form]`, any file field promotes it to `[Multipart]`.
File paths are colour-coded by whether they resolve and are readable
(relative to the collection's directory, matching where Hurl looks). A
`Base64 File` field is encoded at send time behind a configurable prefix, so
`data:image/png;base64,` yields a ready-made data URI.
- **Asserts and captures can be built from a response.** With a reply on
screen, `a` in the Response pane opens a two-step palette: pick a value the
server actually sent — status, duration, any header, any value in the JSON
body, listed beside what it currently is — then pick what to say about it.
The rows are the Hurl lines themselves (`jsonpath "$.data.token" == "ey…"`),
so what you choose is what gets written. Typing narrows the list, and
anything selected in the body pre-fills the filter. The last row on every
value is *keep it in a variable*, which adds a `[Captures]` row under a name
taken from the field itself — the fastest way to chain one request into the
next. Choosing the status sets the `HTTP <status>` line rather than adding a
competing assert.
- **The request preview substitutes `{{ VAR }}`** and colours each by status —
green loaded, cyan literal, orange loading, red missing — while the editor
keeps the original text. Secrets are masked as eight dots.
- **A JSON body can be laid out again** — `p` in the main view, `Alt+P` in the
request editor, **Format** in the GUI. The body is re-indented in place
rather than round-tripped through a JSON parser, so comments, a bare
`{{ TEMPLATE }}` standing where a value goes, number spelling (`1.50`, `1e3`,
a 19-digit id) and duplicate keys all survive; a body that isn't JSON is
refused rather than mangled.
- **Captures are visible in two places, because there are two questions.** The
Response pane's **Captures** tab lists what *that* request captured when it
last ran, marking a value a later run has since replaced as *superseded*. The
live pool — what `{{ VAR }}` is worth *right now* — is in the terminal UI's
`v` **Variables** popup and the GUI's Environments panel ▸ **Variables** tab.
Both list the bound environment's variables and the capture pool together, in
the precedence substitution applies: the environment *overridden by* the pool,
so an environment row a capture is shadowing is marked as such rather than
quietly showing a value that isn't the one being sent. Values are masked by
default (`m` / **Reveal**) since a capture is usually a token; copying still
yields the real value, and `# [Gen]` computed values are never listed.
- **Sections start empty** and dropdowns only auto-open on an empty cell, so
arrowing through a populated table doesn't keep reopening them.
- **Settings ▸ Preferences** persists: confirm on exit/clear, confirm before
deleting an environment or a request, always-save-when-prompted, whether
`Alt+F5` runs the collection in batch mode (chaining cookies and captures),
whether Esc discards request edits without asking (off by default), and the
default Request view (JSON or Hurl).
- **Settings ▸ Theme** ships three presets — Britannia, Parisian Purple,
Dannebrog — one per UI language (English/French/Danish), and follows the
language until you pick one by hand. `Ctrl+N` clones a preset into an
editable custom theme; `Enter` on a colour opens an RGB picker that previews
live and auto-saves. `Ctrl+D` deletes a custom theme.
- **Saving.** **Save** overwrites the file the tab came from without
confirmation; **Save As…** always prompts, and confirms an overwrite. Every
File-menu item has a bracketed mnemonic that both selects and activates it.
## Graphical UI
`paperboy -g`, from a build with the `gui` feature. Feature-for-feature
equivalent to the terminal UI — same tabs, folder tree, request editor,
response viewer, environments panel, reports, theme editor, git remotes, and
the same three languages. What differs:
- **Panels and result columns are resized by dragging**; double-click a column
border to hand it back to the automatic fit. Hand-set widths persist as long
as the report keeps producing the same columns.
- **`Tab`/`Shift+Tab` cycle panels** in the terminal UI's order. The focused
request list is arrow-driven: `Home`/`End`, `Enter` to run, `F2` to rename,
`Delete` to delete, `Ctrl+Z` to undo. A Workspace tree adds `Left`/`Right` to
collapse and expand, and `PageUp`/`PageDown` for ten rows.
- **Single-letter shortcuts don't carry over** — in a desktop window those keys
are text. Globally: `F5`/`Ctrl+Enter` run, `Ctrl+S`/`Ctrl+Shift+S` save,
`Ctrl+W` closes, `Ctrl+Z` undoes a delete, `Alt+F` opens the File menu, `F1`
shows every shortcut.
- **Asserts and captures are built by right-clicking the response.** Click a
value in the body and choose *Assert this…* — the field under the caret is
worked out from the raw JSON, so it works on a minified body as well as a
pretty-printed one. Right-clicking a header row does the same for that
header, and the **Assert…** button beside Copy opens the same builder on the
whole list of values the reply carried. The list is filterable by name or by
value, and "keep it in a variable" adds the `[Captures]` row.
- **The File menu is grouped by verb** (New / Import / Open / Save). Open ▸
Collection and Load ▸ Environment take Postman exports too — they work out
what the file holds. Every dialog reopens where you left it.
- **Workspaces are editable in place**: New adds a collection, report or
environment; drag files and folders onto another folder to move them, or onto
the empty space to move them back to the root. Nothing escapes the workspace
root and nothing is silently overwritten.
- **Reports bind to their collection by a relative path** (`../apis/billing.hurl`
included), so a workspace survives being zipped up or handed over. The
`collection` dropdown offers the report's own workspace first and hides
outside collections behind a toggle.
- **The window remembers itself** — size, every splitter you dragged, the open
view, the selected report/request and the Workspace node.
### The PaperTrail block editor
Reports get a **Blocks** view alongside **Source** and **Results**: a
drag-and-drop editor where blocks are dragged from a palette, reordered, nested
inside `FOR` loops (which move as one, body included) and dropped on the trash
bar to delete. The drag outline and the drop marker are both drawn as the
block's own silhouette, at its real width and indent.
Editable on the blocks: the request a step runs, its alias, response format and
`SHOW(…)`/`HIDE(…)`/`STATISTICS(…)` lists; a `FOR` loop's binder, source, roles
and `PARALLEL(n)` concurrency; and the report's own settings — `collection`,
`output`, `environment`, `root`, `baseline`, `columns` — in a boxed panel at the
top of the flow. Those apply to the report rather than running as a step, so
they're deliberately not blocks. `output` names a *format* (`csv`, `json`,
`html`, `xlsx`), not a filename; only the CLI's `-o` takes a path. Everything has
hover help, and **Source** is highlighted with the terminal UI's colours,
underlining whatever the parser rejected.
### Desktop icon on Linux
Wayland has no per-window icon protocol, so shells match the window's app id
against an installed `.desktop` file. The first GUI launch writes
`$XDG_DATA_HOME/paperboy/paperboy_logo.png` and
`$XDG_DATA_HOME/applications/paperboy.desktop` (with `StartupWMClass=paperboy`
for X11) if they aren't already there, and never touches them again — so you
can customise them. The shell may need a rescan (log out, or restart it) to
notice. Delete both and relaunch to regenerate, which is also how you refresh
`Exec=` after moving the binary.
## Reading a report's results
A finished — or a still-running — report shows its rows in a **Results** grid,
with a filter bar, a summary and, for a ground-truthed report, a confusion
matrix. Both front-ends behave the same way:
- **Sorting is by column.** Click a header in the GUI, or press `o` on a column
in the terminal UI: ascending, then descending, then back to run order.
Numbers sort as numbers, text naturally (`img2` before `img10`), and empty
cells always sink to the bottom. It works mid-run too — rows still on their
way have nothing to sort by, so they wait at the bottom. Sorting is a view:
exports keep the run's order.
- **A single row can be re-run**, which is what a transient failure asks for —
a dropped connection, a momentary 503 — where re-running the whole report is
the only other fix, and where in a comparison the bad row reads as a
regression that never happened. Right-click the row and choose **Re-run this
row** in the GUI, or press `x` on it in the terminal UI. Only that row's loop
iterations go again (both sides of a comparison row), with the parameters the
result was run with, and the fresh answer replaces the old one in place —
`Result`, `Correct`, `Trend`, the statistics footer and the error list
included. A re-run uses the report *as it was when the result was produced*,
so editing the script afterwards doesn't strand the result. During a run, a
row that has already come back re-runs straight away beside the run; while a
re-run is going, further rows are queued (↻) and go as the next batch.
- **Stopping a run keeps everything that came back** — rows, summary, matrix,
filters and sorting — and the run can be **resumed**: the **Resume** button
beside Run in the GUI, or `r` on the stopped run in the terminal UI (`y`
resumes, `n` starts again from scratch). Resuming runs only the rows that
hadn't come back, into their places, and finishes the report as a whole run
would. A stopped run's partial rows can't be re-run individually; resume it
instead.
- **Filters and metrics are live.** Differences, Correct, Incorrect,
Regressions, Improvements and the confusion-matrix cells (`/` in the terminal
UI) all select from the rows that have come back, judged as the finished
report will judge them. Accuracy, Fixed/Regressed and every matrix cell are
there from the first frame and fill in as rows arrive. A case with only one
side back yet isn't counted as a difference — it has its own **Waiting for
pair** filter while the run goes.
## Environments and secrets
A `.vars` file is one `KEY=value` per line. Values can be:
| Literal | `USERNAME=demo` | — |
| Process env var | `BASE_URL={{ env:DEMO_BASE_URL }}` | The process environment |
| 1Password | `API_TOKEN={{ op://Vault/Item/field }}` | The local `op` CLI |
| AWS SSM | `DB_PASSWORD={{ ssm:/path/to/param }}` | Local AWS auth |
Provider references resolve in the background at load time, and the resolved
values are never persisted — `state.json` keeps only the reference. Every
1Password reference across every open collection resolves in a single `op
inject` call, so you get one authorization prompt rather than one per
collection. `r` in the Environment panel retries a single failed entry. Editing
a value into something that looks like a reference triggers a load attempt, and
a "still secret?" checkbox decides whether the new value stays masked.
**Loading a `.vars` file substitutes nothing on its own.** It only joins the
Global Environments list. It then has to be *activated on a tab*: `a` in the
Global Environments panel (GUI: the **Active** button, a double-click, or
right-click → Activate).
The environment belongs to the tab, not to the app: one per tab, and activating
one leaves every other tab alone. Open the same collection in two tabs to run it
against staging and prod side by side. A collection still showing raw
`{{ VAR }}`, or a red "variables in this request are undefined" band, nearly
always means this step was missed on *that* tab.
A variable that is *defined but empty* is not undefined and warns about nothing
— it substitutes as an empty string. With Basic Auth that produces a
well-formed request that comes back `401`.
## Generated values
Some values can't be written down: a nonce, a timestamp, an HMAC over the two.
Postman uses a pre-request script; PaperBoy uses a `# [Gen]` block of named
expressions, evaluated immediately before the request is sent.
```hurl
# [Gen] 3
# nonce = random_hex(16)
# ts = timestamp
# sig = hmac_sha256_b64(API_SECRET, concat(nonce, ts))
POST https://api.example.com/orders
X-Nonce: {{nonce}}
X-Timestamp: {{ts}}
Authorization: HMAC {{sig}}
```
The expressions stay in comments and the request refers to results as ordinary
`{{name}}` placeholders, so the file remains a plain `.hurl` file: stock `hurl`
parses it byte for byte and runs it given `--variable nonce=… --variable sig=…`.
Nothing else could work — Hurl reads a placeholder only as far as the first
character outside `A-Za-z0-9_-` and discards the rest silently, so
`{{ hmac_sha256(K, M) }}` would be sent as the value of `hmac_sha256`. PaperBoy
now refuses to save such a placeholder rather than let it truncate.
The block may sit above the request line, as here, or immediately below it;
both are read. PaperBoy writes it below when it saves, so a hand-written file
in the other order moves its block down the first time it is saved and is
otherwise unchanged.
A bare identifier is a variable reference — an environment variable, a request
parameter, or an earlier row in the same block. Calls nest. Rows are evaluated
in order and a row may only refer to one above it. Values are computed per run,
never previewed, and never written to `state.json`; a secret read through
`{{ op://… }}` is no more exposed by signing with it than by sending it.
What a block computes stays available to the rest of the session, exactly as a
`[Captures]` value does: sign a request, and the request after it can echo the
same `{{nonce}}` — including when you run it on its own. (Memory only, for the
reason above: a fresh PaperBoy computes fresh values.) A `[Captures]` row of
the same name is the later, more specific statement and wins.
One request per name, though. "Run All" and `paperboy -c` normally run one
request at a time, so each block is evaluated in its own window and two
requests may each have their own `nonce`. A **batch** run (the `--batch` flag,
or the Run All batch preference) is a single Hurl call over the whole file with
one variable set, so there the two share the first request's value — a
signature computed over another request's nonce. Both front-ends say so before
starting such a run, and `--batch` prints the warning too; the fix is usually
to not use batch.
The same applies to a name the environment already defines. Running one request
at a time, a block's value overrides the environment's from that request
onwards; a batch has one variable set for the whole file, so it cannot override
from partway through without changing what the *earlier* requests send. Batch
therefore leaves the environment's value in place and says which names it
did that to.
`counter` counts within the process, not within a run: it starts at 1 the first
time it is evaluated and keeps going for as long as PaperBoy is open, so sending
the same request three times gives 1, 2, 3. It is a sequence, not a setting, and
is not saved — a restarted PaperBoy counts from 1 again.
Edit the block in the request wizard's **Generated** section (`Alt+0`), in the
GUI editor's **Generated** tab, or as text. Both editors offer the functions as
you type — with their arguments named — and the GUI's **Function…** menu lists
them all; either way the call is written at the caret, over any part-typed
name, with the caret left between the brackets. Both say what is wrong with a
row while
it is still a typo rather than leaving it to be a 401: an unknown function, the
wrong number of arguments, an expression that doesn't parse. Placeholders that a
generator will fill render in the theme's *generated* colour and keep their
braces, because the value doesn't exist yet.
| Time | `timestamp`, `timestamp_ms`, `iso8601`, `date(fmt)` (strftime, UTC) |
| Random | `uuid`, `counter`, `random_int(lo, hi)`, `random_hex(n)`, `random_alnum(n)`, `random_base64(n)` |
| Encoding | `base64`, `base64url`, `base64_decode`, `hex`, `urlencode`, `urldecode`, `json_string` |
| Hashes | `md5`, `sha1`, `sha256`, `sha512` |
| MACs | `hmac_sha1(key, msg)`, `hmac_sha256`, `hmac_sha512` |
| Text | `concat(…)`, `upper`, `lower`, `trim`, `split(text, sep, n)`, `regex(text, pattern)` |
| JSON | `jsonpath(text, path)` |
| Request | `method`, `url`, `path`, `query`, `header(name)`, `body`, `request_name` |
`jsonpath(text, path)` reads a value out of a JSON document the block already
has in hand — this request's own `body()`, or a response an earlier request
captured whole. When the value comes straight from a response a `[Captures]`
row is the right tool; this is for the cases a capture can't reach, which is
anything that has to be *computed* from the value: signing part of a payload,
or building this request's body out of pieces of the last one's. It walks `$`,
`.name`, `["name"]`, `[n]` and `[?(@.key == 'x')]` — the last of which is how
you address an API that returns its fields as a list of key/value objects. A
string comes back as its text (not with the quotes still on), an object or
array as compact JSON, and `null` is an error rather than the four characters
`null`. Wildcards, recursive descent, slices and unions are refused by name
rather than half-implemented, pointing you at the `[Captures]` row that has
Hurl's full JSONPath: the same path meaning two different things in one
request is worse than not being able to write it.
Every hash and MAC returns lowercase hex — matching `sha256sum` and CryptoJS's
`.toString()`, so a ported Postman script lands right — and each has a `_b64`
variant returning standard padded Base64 and a `_b64url` variant returning the
URL-safe alphabet without padding — the encoding a JWT segment is made of,
where `+`, `/` and `=` are all wrong. The encoding is in the name rather
than a default because a signature in the wrong one is the right length,
entirely plausible to look at, and rejected with the same `401` as a wrong
secret. Note that `base64(sha256(m))` is *not* `sha256_b64(m)`: the first
encodes 64 hex characters, the second the 32 bytes they spell.
The **Request** functions read the request the block belongs to — the method as
sent, the body as it goes on the wire (no JSON comments, no switched-off
headers) — which is how a signature over "the thing I am about to send" is
written. They read the text as authored, substituted against the rows above
them: a row reading `body()` sees earlier rows filled in and later ones still as
`{{name}}`, so a value can never depend on a row that depends on it. Without a
request behind the block — the editor's live check on a row you are still
typing — they say so rather than answering with nothing, because an HMAC over a
silently empty body is a signature that authorises nothing.
`split` counts pieces from the end when given a negative index, so the last
segment of a path is `split(path(), "/", -1)` — JavaScript's `.pop()`, which is
the shape these scripts are written in. `regex` is the escape hatch for what
`split` can't reach: the first capture group if the pattern has one, otherwise
the whole match. Both treat "no such piece" and "matched nothing" as faults
rather than an empty answer, since that text goes on to be signed or sent.
**A name is written bare, not in braces.** Everywhere else in PaperBoy a
variable is `{{name}}`; inside a generator expression it is just `name`, because
an expression already names things — `concat("Bearer ", TOKEN)`. Writing
`"{{TOKEN}}"` there is refused rather than accepted as a string, since a
signature over the eight characters `{{TOKEN}}` is the right length, entirely
plausible, and rejected with the same `401` as a wrong secret. (`\{` is the
escape, for a string that really does want a brace.) A row can also be a plain
literal — `expected = "APPROVED"` — which is how an assert compares against a
per-request expectation: `jsonpath "$.status" == "{{expected}}"`.
**Canonicalisation is yours.** PaperBoy signs exactly the bytes you assemble; it
will not build a canonical request from the live headers, so AWS SigV4 and
friends are out of scope. Chaining a MAC into the *key* of the next one isn't
expressible either, since every value here is text.
The block works headlessly too. `paperboy -c …` evaluates each request's rows
in its own window, so a generator can read a value an earlier request captured
and two requests each get their own nonce. `--batch` is a single Hurl call over
the whole file and has no such window: there every block is evaluated once
before the run, a name computed by two requests takes the first one's value for
both, and the run says so before it starts.
A row that fails — unknown function, wrong arity, a name nothing defines —
reports rather than blocks the send. It binds nothing, so `{{sig}}` goes out
literally and comes back a loud `401`, which is easier to diagnose than a
refusal.
Importing from Postman maps the dynamic variables that have an exact equivalent:
`$guid`/`$randomUUID` and `$isoTimestamp` become Hurl's own `{{newUuid}}` and
`{{newDate}}`, while `$timestamp`, `$randomInt` and `$randomAlphaNumeric` become
`[Gen]` rows. The rest
are renamed and listed in `CONVERSION-NOTES.md` as values you must supply —
guessing at `$randomFirstName` would send a plausible wrong value, which is
harder to notice than a request that won't run.
Worked examples — collections to import, and the `.hurl` file they should become
— are in [`examples/postman/`](examples/postman/).
## Git remotes
Load and save collections, environments and whole workspaces straight from a
remote, with **no local clone**: PaperBoy lists refs, fetches just enough
history to read the file tree, and checks out only the files you actually
asked for. Nothing else in the repo touches your disk, however large it is.
**Loading** (File ▸ Load ▸ *kind* ▸ From Git…): give the URL — `https://…` or
`git@…`, with an optional access token used only for that fetch (GitHub-style
`https://x-access-token:<token>@host/…` is handled for you) — then pick a ref
and a file, both filterable as you type. `↓` on the URL field offers your
recent URLs. Loading a collection then offers to pair an environment from the
same listing, with no second round-trip. Anything loaded from git shows a ⎇ in
its tab title and remembers its origin.
**Workspaces** ask which files to fetch first — `.hurl` and `.json` (default),
`.hurl` only, `.json` only, or everything — and then whether to keep the
download temporarily or copy it somewhere permanent immediately.
> **A temporary workspace is never cleaned up.** Its files live in a temp
> folder for as long as the tab exists — including across a close and undo, and
> across restarts. They accumulate. Choose "save to a permanent location" when
> asked, or later via File ▸ Save ▸ Workspace ▸ Save As…, which copies the
> folder and stops tracking it as temporary.
**Saving** (File ▸ Save ▸ Collection ▸ To Git…) pushes a commit directly to the
remote. The URL is prefilled from where the collection came from, so you can
redirect it to a fork. You choose the in-repo path, whether the attached
environment goes in the same commit, and a branch or tag:
- A **branch** defaults to the one you loaded from, so `Enter` just appends a
commit. `↓` lists the remote's branches. No merge or rebase is attempted — a
non-fast-forward is reported as an error.
- A **tag** must be new. The remote is re-fetched immediately before the check,
and an existing tag is always rejected with no way to force it.
The message defaults to `Update <name> via PaperBoy` and is editable. The author
is your git identity, or `PaperBoy <paperboy@localhost>` if you have none. A
branch push updates the remembered origin and clears the modified markers; a tag
push clears the markers but leaves the origin on your working branch.
To Git… only works for something loaded from git. For anything else, Save As…
into your own clone and use git normally.
## Importing from Postman
Already have an export? Just open it — **Open ▸ Collection** and **Load ▸
Environment** both work out what the file holds, and File ▸ Import from Postman
▸ *From an exported file* says so explicitly. No API key, no account.
To pull from an account, File ▸ Import ▸ Postman account… (terminal) or File ▸
Import from Postman ▸ From my Postman account… (GUI). Give it an API key, pick a
workspace, choose what to bring and where, and the result opens as a workspace.
Paste a workspace id — or its Postman address — on the first step to skip the
listing entirely.
Migrating off Postman altogether: `Ctrl+A` on the workspace list, or **Import
all** in the GUI. Everything the list is *showing* is imported (so the filter is
honoured), each workspace into its own folder, so two "Billing API" collections
from different workspaces both survive.
Postman rate-limits its API, so the wizard shows what it found and roughly how
long the download will take before fetching anything, then reports the remaining
time from the rate it is actually achieving and says when it is pausing to stay
inside the limit. A Postman API key carries its owner's full access and can't be
scoped, so a missing workspace is one your account isn't a member of.
The same import runs headlessly:
```sh
export POSTMAN_API_KEY='PMAK-…'
paperboy --postman-import # list workspaces
paperboy --postman-import --postman-workspace 12ece9e1-… -o ~/API
paperboy --postman-import --postman-all -o ~/Postman # every workspace
```
| `--postman-key` | The key, instead of `$POSTMAN_API_KEY`. Takes the same `{{ … }}` provider references as a `.vars` file — `'{{ op://Private/Postman/credential }}'` keeps it out of your shell history. Never written to disk, stripped from error messages. |
| `--postman-all` | Every visible workspace, each into its own folder under `-o`. Empty workspaces are skipped and inaccessible ones reported rather than fatal, so forty workspaces aren't stopped by one. Excludes `--postman-workspace`. |
| `--postman-what` | `collections`, `environments` or `all`. |
| `--postman-format` | `postman` (default) keeps the JSON byte for byte; `hurl` converts. |
| `--overwrite` | Replace a non-empty destination, which is otherwise refused. |
| `--postman-base-url` | Another tenant, e.g. `https://api.eu.postman.com` for EU Enterprise. |
The result is a folder of `Collections/` and `Environments/`; open it with Open
▸ Workspace.
### Converting to Hurl
`--postman-format hurl` brings across requests, folders (as `Folder/Name`
titles), headers, query parameters, raw bodies and form/multipart fields, plus:
- **Auth, including inheritance.** Collection- and folder-level auth is applied
to requests that don't set their own, and `noauth` opts back out. `basic`,
`bearer` and `apikey` (header or query) are mapped.
- **Collection variables**, which have nowhere to live in a `.hurl` file, as
`<name> (collection variables).vars` beside the environments.
- `pm.<store>.set("NAME", body.a.b)` calls in test scripts, as `[Captures]`.
- **Dynamic variables.** `{{$guid}}` and `{{$isoTimestamp}}` become Hurl's own
`{{newUuid}}`/`{{newDate}}`; `{{$timestamp}}`, `{{$randomInt}}` and
`{{$randomAlphaNumeric}}` become [generated values](#generated-values). The rest
are listed as values to supply.
- **Pre-request scripts**, as far as they reduce to values PaperBoy can compute:
`pm.environment.set("id", uuid.v4())`, `Date.now()`,
`Math.floor(Date.now() / 1000)`, `new Date().toISOString()`,
`pm.variables.replaceIn("{{$guid}}")` and literals become
[generated values](#generated-values).
- **Test scripts**, as the status and assertions they always make:
`pm.response.to.have.status(400)` becomes the request's expected status, and
`pm.expect(...)` checks on the body, headers and response time become
`[Asserts]`. A deep equality against a literal document
(`.to.eql({ id: 7, name: "Ada" })`) is written out one leaf at a time, since
Hurl has no predicate that takes a document — which also makes a failure name
the field that differed. Only checks that run *unconditionally* are taken —
anything inside an `if`, a loop or a helper function is left for you, since an
assertion that was meant for one branch fails every run.
- **`setNextRequest`**, as a note saying which of the four things it was doing:
polling (which Hurl writes as `[Options] retry`), an order you can write down
in the file or as `REQUEST` lines in a
[PaperTrail flow](#the-papertrail-block-editor), a run that stopped early, or
a request name built as the script ran. PaperBoy runs a
collection in file order, so none of them convert — but they are four
different problems with four different fixes.
- **Scripts on a folder or on the collection**, which Postman runs for every
request inside; they are converted for each request they cover, and reported
once against the folder that holds them.
Hurl doesn't cover everything Postman does. Anything dropped — the rest of a
script, OAuth 2, GraphQL bodies — is listed per request in
`CONVERSION-NOTES.md` at the root of the import; no file means nothing was lost.
A collection this build can't read is written out as its original JSON, so
converting can't cost you data.
## Headless runner
```sh
paperboy -c collection.hurl
paperboy -c collection.hurl -e environment.vars
paperboy -c collection.hurl --batch
```
`-c` takes a `.hurl` file or a Postman export. `-e` supplies the environment.
Exit status is `0` only if every request passed.
By default each request's method, URL, status, asserts, captures and truncated
body print as it finishes, coloured unless the output isn't a terminal or
`NO_COLOR` is set. Streaming runs one request at a time through the same `hurl`
runner, so captures still chain — but it can't carry Hurl's automatic cookie jar
between requests, and says so at startup. An explicit `[Cookies]` section is
unaffected. `-b`/`--batch` runs the collection as a single Hurl call, trading
incremental output for cookie continuity.
### Reports
`-r report.trail` runs a PaperTrail report and exits.
```sh
paperboy -r report.trail # collection from the report's headers
paperboy -c api.hurl -r report.trail -o out.csv # or given explicitly; - is stdout
paperboy -c api.hurl -e prod.vars -e staging.vars -r report.trail
paperboy -c api.hurl -r report.trail --dry-run # expand it, send nothing
paperboy -r report.trail -o out.html -o out.json # one run, several formats
paperboy -r report.trail --param CASES_DIR=./batch-07 # set a PARAM the report declares
```
Without `-c`/`-e` the report's own `# collection:` / `# environment:` headers
apply, resolved relative to the report. `-e` is repeatable: each file is named
by its stem and becomes selectable in an `ENVS` loop, so `-e prod.vars -e
staging.vars` satisfies `FOR … IN ENVS BASELINE("prod"), COMPARISON("staging")`;
the first is also the base variable layer. `-o`'s extension picks the format
(`.csv`, `.json`, `.html`, `.xlsx`, `.pdf`, or `.xml` for JUnit), `-` writes CSV
to stdout, and
omitting it derives the filename from the report's own headers.
`-o` is repeatable, and every file comes from **one** run of the requests — the
report is rendered once per format from the same result, never run twice. That
is what an application embedding PaperBoy needs: `-o out.html -o out.json` gives
it a rendering to show a user and a structure to parse, with no risk of the two
disagreeing because they came from separate runs. `-` may be given at most once
(two formats down one pipe would interleave into neither), the same path twice
is refused as a typo, and every format is checked *before* any request is sent,
so a misspelled extension costs nothing. If one file fails to write, the others
are left in place and the run exits `1`.
`--param NAME=VALUE` supplies a value for a `PARAM` the report declares, and is
repeatable. It is what lets one report serve many runs: a report that declares
```
PARAM FOLDER CASES_DIR = "./cases"
FOR CASE IN FOLDERS "{{CASES_DIR}}"
```
is pointed somewhere new with `--param CASES_DIR=./batch-07` rather than by
editing the file — which is never rewritten, so a report under version control
keeps meaning the same thing to everyone. The value beats the declared default,
is checked against the declaration (a `CHOICE` must be one of its options, a
`NUMBER` must parse), and a name the report doesn't declare is an error rather
than a value that silently does nothing. A `PARAM` with no default *requires*
a `--param`, since there is nothing to fall back on.
Exit codes are a contract for callers: `0` ran clean, `1` a setup error or a
run in which something failed — a request that did not come back, an assertion
that did not hold — `3` some steps were skipped because something they depended
on failed, `4` the run was stopped before it finished (see [Stopping a
run](#stopping-a-run-ctrl-c---grace---stop-on-stdin)), `5` a `--fail-under` or
`--require-net-gain` gate was not met, `130` a stop that was repeated and so forced, and `2`
(clap's) means the command line itself was wrong. Progress goes to stderr, so
`-o -` leaves stdout clean for a pipe.
#### In a pipeline (`--fail-under`, `--require-net-gain`, JUnit)
A report is a deploy check as much as it is a document, and a pipeline reads
one thing: the exit code. Two failures are worth telling apart.
*The run broke.* A request that never came back, or that failed its
assertions, fails the run — exit `1`, the same as `paperboy -c`. The row still
appears in the report with its error in the cell, because the point of a report
is to show every case rather than stop at the first, but the process does not
claim success.
*The answers were wrong.* Everything was sent and read, and the API's answers
did not match the ground truth. That run is faultless by every measure above,
and `--fail-under` is what turns it into a failed pipeline step:
```sh
paperboy -r post-deploy.trail --fail-under 95 -o results.xml
```
The gate is measured on the whole run's `Correct` roll-up — the per-row verdict,
over the rows that had a `TRUTH` to compare against — and fails the run with
exit `5` when it comes out below the threshold. Rows with no ground truth are
not counted as wrong: they were never asked. The comparison is made on the
unrounded figure, so `--fail-under 100` means *every scored row was right*
rather than "right to one decimal place"; where the rounded figure would
contradict the verdict, the summary prints both numbers to the precision that
separates them (`accuracy 94.96% — UNDER the required 95.00%`). A report that
scores no column at all has nothing to gate on and is refused *before* anything
is sent, rather than passing every deploy on no evidence; `--dry-run` and
`--postman-import` are refused for the same reason. If a run that declared a
`TRUTH` nevertheless scores nothing, that is a broken run and exits `1`: there
is no accuracy to hold to a threshold. A run that was **stopped** (exit `4`) is
not given a gate verdict at all — the figure would be drawn from whichever rows
happened to finish. `5` is its own code because the two failures call for
different responses: `1` is worth retrying, `5` never is.
*The answers got worse.* An accuracy floor is an absolute line, and a suite
that has climbed well above it can break rows for months without ever reaching
it — a run at 98% that breaks three rows in a hundred still clears
`--fail-under 95` comfortably. `--require-net-gain` gates on the *change*
instead, against the baseline the report already compares with:
```sh
paperboy -r nightly.trail -e prod.vars -e staging.vars --require-net-gain 0
```
It counts rows the way the `Trend` column does — rows fixed minus rows
regressed, over the rows scored on **both** sides — and fails the run with exit
`5` when that comes out below the number given. `0` is "do not go backwards",
which is the usual one and needs no maintenance as the suite improves; a
positive number demands progress, and a negative one is a tolerance
(`--require-net-gain -2` lets two net regressions through). Trading a fix for a
regression passes at `0`; a run that wants to hear about churn asks for `1`.
The two gates can be used together and share exit `5`: either one failing is
enough. `--require-net-gain` needs both halves of the comparison it is named
after — something to be right or wrong about (a `TRUTH`) and something to have
been right or wrong about last time (an `ENVS BASELINE(…)/COMPARISON(…)` clause
or a `# baseline:` snapshot) — and a report missing either is refused before
anything is sent, exactly as `--fail-under` is. A run that was stopped, or one
where no row reached a baseline with a truth on both sides, gets no verdict:
that is a broken run (exit `1`), not a quality miss.
`-o results.xml` writes **JUnit XML**, which is what makes a run show up as
test results in a CI UI rather than as a file nobody opens: one `<testcase>` per
row, named by the row's key so a case keeps its identity between runs, with a
failed request as `<error>`, a wrong answer as `<failure>` carrying both sides
of the comparison, and a dry run's cases as `<skipped>`. Each case gets exactly
one of those, in that order of seniority — a row whose request failed is an
`<error>` even if its answer was also wrong, because the stricter JUnit schemas
allow a case one outcome and a case counted twice makes every consumer's
arithmetic disagree; the demoted verdict is written into the case's
`<system-out>` rather than lost. Run errors that belong to no row (an empty
glob, a producer that failed before any row existed) and the run's caveats
become one extra case, so a run that produced nothing *because* something was
wrong is never a green suite of zero tests — and so the caveats are visible in
the CI systems that only read output attached to a case. It is an ordinary `-o`, so one run can write
`-o results.xml -o report.html` and give the pipeline its verdict and a human
the detail.
#### Watching a run from another program (`--progress-json`)
`--progress-json` replaces the human `done/total` counter with newline-delimited
JSON on stderr: one event object per line, flushed as it happens, so a caller
reads it incrementally (`for line in proc.stderr` and an ordinary `json.loads`
per line — no streaming parser).
```sh
paperboy -r nightly.trail -o out.json --progress-json 2>events.ndjson
```
```json
{"event":"plan","schema":1,"report":"nightly","dry_run":false,"total":2,"columns":["Case","Status"],"withheld_columns":["Raw: Response"]}
{"event":"row_started","schema":1,"path":"0.0","row_index":0}
{"event":"row_completed","schema":1,"path":"0.0","row_index":0,"ok":true,"target":null,"cells":{"Case":"a","Status":"200"},"errors":[],"withheld":[]}
{"event":"output_written","schema":1,"path":"out.json","format":"json","ok":true,"error":null}
{"event":"run_finished","schema":1,"ok":true,"exit_code":0,"rows":2,"interrupted":false,"dry_run":false,"partial":null,"rows_completed":null,"rows_planned":null,"rows_ok":2,"rows_failed":0,"warnings":[],"skipped":[],"errors":[],"scored":{"compared":2,"correct":2,"incorrect":0,"accuracy":100.0},"movement":null,"gate":null}
```
`plan` arrives once, after the projection pass and before a single request is
sent, so a UI can draw the whole empty grid up front. `row_started` and
`row_completed` fill it — several rows are in flight at once under `PARALLEL`,
which is why every row event carries **`path`**: the row's structural identity
(`"0.3"`, `"0.1.2.0"`, `""` for a report with no loop), assigned before the run
and stable however the workers interleave. Alongside it, `row_index` is the
slot that row occupies in the projected grid — and, for a report that doesn't
collapse an `ENVS` comparison, the index of its row in a `-o out.json` report.
That is what makes the live grid and the finished file explicitly linkable: a
consumer can keep progress cheap and read the full values from the report at
the end. `run_finished` is the last line and carries the exit code the process
is about to use. With `--fail-under` it also carries a `gate` object
(`{"metric":"accuracy","required":95.0,"actual":92.4,"compared":250,
"passed":false,"note":"…"}`), and `null` without one — so a consumer can tell a
gate that passed from a deploy nobody checked.
It also carries the run's own arithmetic, so a caller gating on quality itself
does not have to parse the report it just wrote: `rows_ok`/`rows_failed` beside
`rows` (counted over the rows that actually ran — a projected or never-reached
row is neither), a `scored` object (`{"compared":250,"correct":231,
"incorrect":19,"accuracy":92.4}`, the same unrounded percentage `gate.actual`
reports), and a `movement` object (`{"fixed":3,"regressed":1,"still_wrong":2,
"unchanged":244}`) against the report's baseline. Both are deliberately `null`
rather than zeroed when there is nothing to report: a run with no ground truth
has not scored 0%, and one with no baseline has not held steady. These read the
run rather than the table, so a `# columns:` directive that hides a column
cannot change them. With `--require-net-gain`, `movement` also carries that
gate's verdict — `required_net_gain`, `net_gain`, `passed` and `note` — beside
the counts it was drawn from, rather than in a second place that could drift
from them.
The stream stays cheap deliberately: `DETAIL` and `IMAGE` columns — the ones the
report itself marks as drill-down content, a raw response body among them — are
named in `plan` as `withheld_columns` and left out of every row, as is any cell
over 4 KB (named in that row's `withheld`). Those bytes are already being
written to the report file; streaming them per row as well would make progress
cost more than the run.
`schema` is on every event so a consumer can refuse a stream it doesn't
understand rather than mis-read it. The human progress counter and summary are
suppressed rather than interleaved (the summary only when it would share stderr,
i.e. with `-o -`), so the stream is nothing but events.
`run_finished` is unconditional. A fatal *setup* error — an unreadable
collection, a mistyped `--param`, a validation failure — emits no `plan` and no
rows, but it still ends the stream with `run_finished`, `exit_code: 1` and the
diagnostic in `errors`, so a consumer always has exactly one terminal event to
wait on and never has to parse prose to find out what went wrong. (Only a
command-line *usage* error is prose: clap rejects it and exits 2 before PaperBoy
runs at all.)
`--dry-run --progress-json` streams the whole grid too — `plan` and a
`row_started`/`row_completed` pair per projected row, cells and all — without
sending a request. It is the cheap way for a consumer to draw the grid, or to
check its own parsing, before committing to a real run.
A dry run's report file is a **plan, and says so**. Its rows are the ones the
run would produce with the response columns blank, so every format carries the
caveat in the same idiom a stopped run's `PARTIAL` uses: a `DRY RUN` banner
above the HTML, `"dry_run": true` at the top level of the JSON (and on
`run_finished`, beside the existing key on `plan`), a final `DRY RUN,…` record
in the CSV, a bold row under the XLSX summary, the PDF title. Nothing in it is
scored either: a ground-truth verdict reads `untested` rather than `incorrect`,
a comparison `Result` reads `not run (dry run)` rather than claiming the
candidate matched its baseline, and there are no metrics at all — a preview has
no accuracy, not an accuracy of 0%.
#### Stopping a run (Ctrl-C, `--grace`, `--stop-on-stdin`)
An interrupted run still writes what it has. Press `Ctrl-C` (or send `SIGTERM`)
during a report run and PaperBoy winds down rather than dying:
1. **No new rows start.** The switch is read where a row is *claimed*, so the
rows already in flight — up to five live HTTP calls under `PARALLEL(5)` —
finish whole rather than being abandoned mid-upload.
2. **`CLEANUP` still runs.** This is the one that would otherwise bite:
`CLEANUP` exists to delete sessions and release locks, and an interrupt that
skipped teardown would leak them against a real service. Every block returns
on its ordinary path, so teardown happens exactly as it would have.
3. **The requested outputs are written, marked partial.** Not "nothing to
report" — a four-hour run stopped at three hours is still three hours of
answers. Every format says so in its own idiom: a `PARTIAL` banner above the
HTML, `"partial": true` with `rows_completed`/`rows_planned` at the top level
of the JSON, a final `PARTIAL,37 of 120 rows ran` record in the CSV, a bold
row under the XLSX summary, and the PDF title. A report that silently looked
complete while covering 60% of the corpus would be a trap.
4. **Exit `4`.** "I was told to stop" is not "your API is broken" (`1`) and not
"steps were skipped" (`3`).
5. **`--progress-json` still ends with `run_finished`**, carrying
`interrupted: true`, `partial: true` and the two counts, preceded by a
`run_stopping` event naming the `source` of the stop and the
`grace_seconds` it allows. An interrupt never reduces to a silent EOF — and
nothing follows the terminal event either: a row that lands after a run has
been given up on is not announced, because a consumer that finalises its
state on `run_finished` would have no way to take it.
```sh
paperboy -r nightly.trail -o out.html -o out.json --grace 60
```
`--grace SECONDS` (default `30`) bounds the wind-down. If rows are still in
flight when it expires, PaperBoy gives up on them and writes the report from
the rows that *did* finish — a degraded rendering, without the metrics and
ground-truth scoring that can only be computed over a complete run, and
carrying a warning that `CLEANUP` may not have run. Its rows are still in
report order: they are harvested as they *finish*, so under `PARALLEL` they
arrive scrambled, and they are laid back into the report's canonical order
before it is written — a report is the same at any degree of parallelism, and
two stopped runs of the same corpus are diffable. A **second** `Ctrl-C` skips
even that and exits `130` immediately: the escape hatch for a `CLEANUP` hanging
on the very service that stopped responding.
`--stop-on-stdin` adds an explicit, cross-platform stop channel: a line reading
`stop` on stdin begins the same wind-down, and so does EOF. Signals are awkward
on Windows, where there is no `SIGTERM` to send a child process, and a caller
that already reads `--progress-json` from stderr gets a bidirectional protocol
by adding one flag. It is opt-in precisely because EOF counts: most CI runners
start processes with stdin already closed.
#### Dependency graphs
Inside a `GRAPH … END` region the order statements are written in stops being
the order they run in. PaperBoy reads what each request needs and what each
produces, works out the ordering that satisfies those, and runs that. A region
that cannot be ordered — two requests each waiting on the other — is an error,
and nothing in the report is sent: the author has said written order is not the
specification, so falling back to it would be the one answer guaranteed to be
wrong.
```
GRAPH release
REQUEST auth/login AS login # captures token
REQUEST orders/create AS order # uses {{token}}
REQUEST orders/get USING(id = {{order.id}})
END
```
Wrapping an existing block in `GRAPH … END` changes nothing while everything
succeeds: with no edges to reorder by, a region runs in written order. Two
things do change once something goes wrong or is ambiguous, and both are the
point of the region rather than accidents of it:
- **A failure stops what depended on it.** Flat, every later request is sent
regardless; in a region, the steps downstream of a failure are skipped and
the run exits `3`. A request that cannot work without a token nobody
captured has nothing to tell you, and sending it anyway costs a real call
against a real service.
- **An ambiguous capture is an error, not last-writer-wins.** Flat, two
requests capturing `token` are resolved by written order. In a region there
is no written order to fall back on, so a step reading `{{token}}` that two
steps in its region capture is refused. Name the one you mean —
`{{login.token}}` — and it is unambiguous again.
A dotted name in a report's own text always means *step, then capture*. The
namespace is reserved: an environment variable whose key happens to contain a
dot cannot answer one, because it would answer precisely when the step had not
captured — quietly supplying a stale credential at the moment the reference was
supposed to fail. A dotted name whose prefix is not a step in scope is an error,
wherever it is written: a `USING` value, a computed column, or a producer path.
Inference reads every place a value can actually be used: a URL, a header, a
body, a `[Captures]` or `[Asserts]` expression, and the bare identifiers of a
`# [Gen]` row, which are variable references despite having no braces. It does
not read a disabled row or a `[Reports]` field — nothing substitutes into
either, so an edge drawn from one would reorder a region for text that never
runs.
`--targets a,b` runs only the named steps and whatever they transitively
depend on, so a release check can ask for one answer without paying for the
whole report. Naming a step that no region declares is an error rather than a
silent empty run, and so is naming a set of targets that leaves out a step the
rest of the report still refers to: the reference could only reach the run as a
literal `{{create.sid}}`. Only qualified references are checked: a plain
`{{sid}}` is answered by whatever is standing in the capture chain, and pruning
can't know what else might answer it — the environment isn't even loaded at that
point — so a selection that leaves a flat reference's producer out is allowed
through. A `TRUTH` template counts as a reference like any other, wherever it is
written — on a computed column, on a variable column, on a `WITH` field or in
the `columns:` directive. It is resolved against the row's cells, which are
keyed by step, so pruning the step it names would leave every row in that column
scoring `Untested` for no stated reason. A `USING(…)` override is read as part of the request too, since it
decides what is actually sent — it can strand a teardown that looked clean, or
clear one that didn't. `--dry-run` lists the steps grouped by how deep in the graph they sit,
which is how you check the shape of a region without sending anything.
`DEPENDS` states an ordering the data doesn't show. Inference only sees values
flowing from a capture to a reference, and some dependencies leave no such
trace — uploading a file that a later request fetches by an id it already had,
say. `REQUEST dfa/result DEPENDS upload` says so outright. Names are the step
names, separated by commas, and a `DEPENDS` is only meaningful inside a region:
outside one, written order already *is* the order, so PaperBoy rejects it
rather than let it look like it did something.
Clauses may be written in any order, and a long statement may gather them into
a bracketed group opening on the statement's own line:
```
GRAPH
REQUEST dfa/result AS result (
DEPENDS upload, session
USING(query.id = "{{session.id}}")
)
END
```
#### Running a region in parallel
`PARALLEL(n) GRAPH … END` lets up to `n` steps overlap. Each is taken the
moment its dependencies are done — not a wave at a time, which would make the
region cost the slowest step at every depth. A cap is permission, not an
instruction: a chain still runs one at a time however high `n` is set, and the
report is identical at any degree, because rows, columns and errors are merged
in plan order rather than in the order workers happened to finish.
#### Shuffling, and why
A region is a *claim* that its edges — inferred and declared — are the complete
set. PaperBoy cannot verify that claim. It can help you falsify it.
With the default tie-break, ready steps run in written order, so a dependency
nobody declared keeps working by accident and surfaces months later when
something unrelated moves. `--shuffle` picks at random among the steps that are
ready, which turns that into a failure now, and prints the seed:
```
Shuffle : seed 4711 (replay with --shuffle=4711)
```
`--shuffle=4711` replays that run. Shuffling only reorders steps that may
legally run in any order; it never runs a step before what it depends on.
How exact the replay is depends on the degree. A sequential region replays
*exactly*: the seed alone decides every choice. A `PARALLEL(n)` region replays
its dispatch *preferences* exactly, but which step becomes ready next also
depends on which request came back first, and no seed controls the network. So
a shuffled failure in a parallel region is far more likely to reproduce under
its seed than without one, but it is not guaranteed to. If you find one and
want it nailed down, re-run the seed with `PARALLEL(1)` — a missing dependency
is a property of the ordering, not of the concurrency, so it will still be
there.
#### Cleanup
`CLEANUP` marks a request that undoes something — deleting a session, releasing
a lock. It is written where it belongs logically but runs at the end of its
block: at the end of the flow at the top level, at the end of each iteration
inside a `FOR`. Cleanups run in reverse dependency order, so a thing is torn
down before whatever it was built on.
```
REQUEST auth/login AS login
CLEANUP auth/logout USING(header.Authorization = "{{login.token}}")
REQUEST orders/create
```
A cleanup whose dependency never succeeded is skipped — there is nothing to
undo — and a cleanup that fails is reported as a warning rather than an error,
because a teardown failing is nearly always a consequence of the real failure
and shouldn't be allowed to bury it.
"Its dependency" means whatever it names in `DEPENDS`, plus whichever step
actually produced each value it reads. A request that fails can still have
captured, so the step a cleanup is gated on is the one whose value it will be
handed, not the last one that happened to succeed — otherwise a teardown could
be authorised by one step and then sent with a different, failed step's
identifier. Cleanups can depend on each other, by `DEPENDS` or by reading one
another's captures, and are ordered accordingly; a cycle between them is
refused: every member of a ring waits on a member that has not run, so none of
them is sent. A cleanup can only depend on one in its own
block, and can only read one's captures there too: an enclosing block unwinds
after this one, so a cleanup out there could never have run in time. For the
same reason an ordinary step can never read a cleanup's capture — teardown runs
after every step in its block.
Which step a cleanup waits for and which one it is *gated* on are two different
questions. The order has to be settled before anything has run, so it can only
ask which steps declare a name; the gate is asked when the teardown is
dispatched, by which point the step that actually wrote the value it is being
handed is a fact. A sibling cleanup that was skipped, or that ran without
capturing, therefore doesn't vouch for a value it never wrote — the step that
did write it is the one that has to have succeeded.
#### Carrying the requests in the report
A report normally names a collection to draw its requests from. It can instead
carry them itself, in a `REQUESTS` section — plain Hurl, which must be the last
thing in the file:
```
# name: Health check
GRAPH
REPORT REQUEST ping
END
REQUESTS
# ping
GET https://example.com/ping
[Asserts]
status == 200
```
That runs with no `# collection:` line and no sibling `.hurl` file:
`paperboy -r health.trail`. A report that embeds its requests may still name a
collection as well, in which case both sets are available and a name used by
both is an error — a reference has to mean one thing.
Embed when the requests exist only to serve the flow, so that the whole check
travels as one file and nothing can be moved out from under it. Reference a
collection when the requests *are* the API surface under test and other things
use them too. Note what embedding does and doesn't buy: it removes the sibling
collection file, not a fixture directory that `FOR … IN FILES` reads.
#### Exit codes
| `0` | Everything ran and every assertion passed. |
| `1` | Something failed: a request, an assertion, or the report itself. |
| `3` | Steps were skipped because something they depended on failed. |
| `4` | The run was stopped before it finished. |
| `5` | A `--fail-under` or `--require-net-gain` gate was not met: the run was clean, the answers were not good enough. |
`3` implies `1` — a skip only ever follows a failure — and says the run is
additionally incomplete, so a pipeline that only cares about pass/fail can
treat any non-zero code the same way while one that reruns can tell the
difference. `2` is left alone: it is what `clap` uses for a bad command line.