# qex — Queued EXecutor
qex is a job queue for long tasks on one machine. It controls the number of
cores and the quantity of memory that the jobs use together.
qex is for coding agents and for the people who work with them. Several agents
on one machine each start work, and no agent sees the load of the others. The
machine then runs out of memory. qex gives those agents one queue.
```sh
ID=$(qex submit --cpu guess --mem guess -- uv run train.py)
qex wait $ID
qex logs $ID
```
**Agents: run `qex help agents` first.** It is one page and it covers everything.
## The three problems that qex solves
**1. No agent sees the load of the others.** Each agent finds free memory,
starts a large task, and the out-of-memory killer selects a victim. qex holds a
budget for the machine. A job starts when the machine has capacity for its
claim, and it waits when the machine does not.
**2. Every hand-rolled watcher waits on a proxy, and a proxy can go permanently
false.** An agent with no way to wait writes a monitor. That monitor watches a
*proxy* for the work — a pattern in the process list, a line in a log, a file
that should appear — and a proxy can stop being reachable without anything
noticing. Three real watchers on one machine, in one day, slept for **54 hours
combined** on conditions that could never become true:
```sh
while pgrep -f "solve.py"; do sleep 60; done # matches its own command line
until grep -q "DONE" run.log; do sleep 60; done # the writer was killed
until grep -q "READY" ~/other.log; do sleep 60; done # that file never existed
```
Only the first is the classic `pgrep -f` self-match. The other two contain no
pattern bug at all — they are careful commands whose evidence simply stopped
arriving. That is the general failure, and it is why the fix is not "write a
better pattern".
qex waits on the **process**, not on a proxy for it. qex is the parent of your
task and calls `waitpid` on that exact process: it exits or it does not, and
there is no third outcome. `qex wait` therefore always returns — including 125
the moment somebody kills the job, where a log-watcher would still be sleeping.
**3. There is no handle on a running task.** qex gives each job a UUID. Use that
id to read the state of the job, read its output, stop it, or remove it from the
queue.
## Install
Take the file for your machine from [the latest
release](https://github.com/stephenc/qex/releases/latest):
```sh
```
Or build it from the source:
```sh
cargo install --path .
```
qex needs Linux or macOS. It has no other requirement. The first command starts
the background coordinator for you, and you do not configure a service.
## Five minutes
```sh
ID=$(qex submit --cpu guess --mem guess -- make test) # gives the id at once
qex list # what operates now
qex status $ID --wait # wait, then the result
qex logs $ID --tail 50 # the output
qex kill $ID # stop it
```
Put `qex run -- make test` in front of a command instead when you wait for it
now: the output arrives as it happens, and the exit code is the exit code of the
job.
A pipeline gives each stage its own log, its own exit code and its own claim:
```sh
BUILD=$(qex submit --name build -- make)
TEST=$(qex submit --name test --needs $BUILD -- make test)
qex wait $TEST # 1 if the test failed, 126 if the build failed
```
## Your session can stop, and the work continues
The job is not a child of your shell, and it is not a child of your agent. A
supervisor holds it in its own session, and the record of the job is on the disk.
Somebody can therefore stop your agent, close the terminal, or replace the qex
binary. The job continues and it still writes its result. Your wait is the only
thing that stops, and any later session attaches to the same job with the id:
```sh
qex submit --id-file build.id -- make # session 1
# a person stops the agent here. `make` continues.
qex status "$(cat build.id)" --wait # session 2, and the result is there
```
**This is why an agent that uses qex is safe to interrupt.** A monitor script
holds the answer in its own memory: stop the monitor, and the answer is gone.
Put the id file where it lasts longer than the session — the project directory
or the home directory, and not a scratch directory that the harness owns or
`/tmp`. qex gives a warning when the file goes to such a directory, because the
file would go away at the moment that the handle becomes necessary.
## What qex does not do
**qex does not limit the CPU of a job.** The `cpu` controller of cgroup v2 is
not available to a user on a usual Linux system, and macOS has no equivalent.
The queue controls the number of cores instead.
**qex does not limit the memory of a job by default.** A claim controls the
queue only. Linux can apply a real limit with cgroup v2; set `[enforce] mode`.
qex never reports a limit that it did not apply.
**A claim is a promise, and not a measurement.** A job that claims 2GB and uses
20GB can still fill the machine. qex tests the free memory before each start,
which limits the damage, but an accurate claim is better. qex measures each job
and uses the measurement for the next job of the same command.
## The documentation
The full documentation is at
[stephenc.github.io/qex](https://stephenc.github.io/qex).
| [Agents](docs/agents.md) | The page for an agent. Start here. |
| [Reference](docs/reference.md) | Each command, option and configuration field. |
| [Design](docs/design.md) | The coordinator, the supervisor and the files. |
| [Security](docs/security.md) | What qex writes, and who can read it. |
Every page is also in the binary, so an agent needs no network: `qex help
agents`, `qex help resources`, `qex help config`, `qex schema status`.
## Development
```sh
cargo test -- --test-threads=2
cargo build --release
```
Each end-to-end test makes its own config and state directory, starts its own
coordinator, and stops it at the end. The tests do not touch the coordinator of
the user, and they turn the peer accounting off.
Use two test threads. Each end-to-end test starts real processes and waits for
them. With more threads, the machine becomes busy, a job starts late, and a test
reports a failure that the program does not have.
The documentation, the code comments, the help text and the error messages use
Simplified Technical English (ASD-STE100).
See [CONTRIBUTING.md](CONTRIBUTING.md). The first line of each commit message
gives the next version number, so its form is part of the build.
## License
Apache License 2.0. See [LICENSE](LICENSE).