pub fn banner() -> String {
format!(
" ==> AGENTS: run `qex help agents` first. It is {} lines, and it is complete.\n\
\x20 It shows how to start a job, wait for the job, and read the output.\n\
\x20 Do not write a monitor script. The command `qex wait` does that work.\n",
AGENTS.lines().count()
)
}
pub const TOPICS: &[&str] = &[
"agents",
"job-file",
"config",
"resources",
"states",
"events",
"output",
"exit-codes",
"pipeline",
"each-line",
"pause",
];
pub fn topic(name: &str) -> Option<&'static str> {
match name.trim().to_ascii_lowercase().as_str() {
"agents" | "agent" => Some(AGENTS),
"job-file" | "jobfile" | "job" => Some(JOB_FILE),
"config" | "configuration" => Some(CONFIG),
"resources" | "resource" | "budget" => Some(RESOURCES),
"states" | "state" => Some(STATES),
"events" | "event" | "stream" => Some(EVENTS),
"output" | "json" => Some(OUTPUT),
"exit-codes" | "exit" | "exitcodes" => Some(EXIT_CODES),
"pipeline" | "pipelines" => Some(PIPELINE),
"each-line" | "eachline" | "fan-out" | "fanout" => Some(EACH_LINE),
"pause" | "resume" => Some(PAUSE),
_ => None,
}
}
pub const AGENTS: &str = "\
qex for agents
==============
Use qex to run a long task. qex holds the task in a queue, starts it when the
machine has capacity, and records the result. You can then wait for the result
with one command.
Do not write a monitor script
-----------------------------
Every monitor that you write waits for a PROXY: a pattern in the process list, a
line in a log file, a file that appears. A proxy can become permanently false,
and nothing tells the monitor. It then waits for ever.
Four monitors were measured on one machine in one day, and together they slept
for 95 hours. Not one of the conditions could ever become true. Three of them:
while pgrep -f \"solve.py\"; do sleep 60; done
The command line of this shell holds the letters `solve.py`, so the
pattern matches the monitor itself. The task stops, one process stays,
and the count never reaches zero.
until grep -q \"DONE\" run.log; do sleep 60; done
Correct, until somebody stopped the task that writes that line. The
marker will never arrive now.
until grep -q \"READY\" ~/other.log; do sleep 60; done
That file was never made. This monitor slept for 41 hours.
A different user found this one later, on a machine that two agents shared. It
had slept for 63 hours:
while true; do M=$(ps -Ao args | grep -c solver)
K=$(ssh other-host 'ps -Ao args | grep -c solver')
[ $M -eq 0 ] && [ $K -eq 0 ] && break; sleep 300; done
A COUNT, and not a test of one process. This monitor waits until nothing
matches. On a machine that two agents share, that condition is not
satisfiable: the work of the other agent holds the count above zero for
ever. The work of this author finished two days before, and the
monitor opened about 750 connections to the other machine while it
waited.
The last three hold NO PATTERN FAULT. They are careful commands. The fault is
the proxy: a log line is evidence of the work, and evidence stops when the work
stops, in a way that the monitor cannot see. The last one is the most dangerous,
because a careful author writes it: on a machine that two agents share, \"wait
until nothing matches\" can never become true.
qex waits for the process, and not for a proxy of the process. qex is the parent
of your task and it uses `waitpid` on that exact process. A process ends or it
does not, and no third condition exists. `qex wait` thus always gives an answer:
the job succeeded -> 0
the job failed -> 1
the job never started, and it
had a `--max-queue-time` -> 123
somebody stopped the job -> 125
a job before it failed -> 126
A task that somebody stops gives the code 125 at that moment. A monitor that
watches a log file would still be waiting.
The same fault applies to every search of the process list. `pgrep -f qex` also
matches the shell command that holds those letters. To find the coordinator, use
`qex info`, which gives the process id from the coordinator itself.
qex watchers
That command finds the monitors of this kind on your machine. It removes its own
process and the processes that started it before it reports anything, so it
never finds itself. A user who looked for this fault with `pgrep -f pgrep` found
the search, and that was the fourth time in one day that the fault appeared.
When to use `qex run`
---------------------
qex run -- make test
Use `qex run` for work that is SHORT AND HEAVY and that you wait for now: a test
suite, a release build, a data conversion. The job goes in the queue, so it
starts when the machine has room, and the other people and agents on this
machine keep the capacity that they claimed. The output arrives as it happens,
on the same two streams, and the exit code is the exit code of the job, or 125
when something stopped the job. Nothing else in your script changes.
WHAT YOU GIVE UP. `qex run` ties the job to this command, but only for the stops
that it can catch:
Ctrl-C stops the job, and not this command only.
A SIGTERM on this command stops the job too.
A SIGKILL does NOT stop the job, and neither does the hangup of
a terminal that closes. The job continues, and `qex list` finds it.
A job that operates receives a SIGTERM. A job that still waits in the queue
leaves the queue instead, because a job with no process cannot receive a signal.
That is correct for work that you are waiting for, and it is WRONG for work that
lives longer than your attention. `qex submit` gives the job a life of its own:
it continues when your session stops, and a later session reaches it with the
id.
short, and you wait for it now -> qex run -- ...
long, or you come back to it -> qex submit, then qex status <id> --wait
When something stops the job, `qex run` gives 125 and not 1. Another agent on
this machine can run `qex kill` or `qex cancel` on your job, because a job of
`qex run` is a job like any other. The code 125 says that something stopped the
job before it could finish, and it does not say that your work failed. Do not
start the work again before you read the line on stderr. Run
`qex help exit-codes` for the full table.
A job that a dedupe key gave you is the one exception. This command did not
start that job, so Ctrl-C stops this wait only, and the job continues. `qex run`
then gives 124, which says that YOUR WAIT ended. Read the section on the key
below.
The three commands you need
---------------------------
ID=$(qex submit --cpu 2 --mem 4GB -- uv run train.py)
qex wait $ID
qex logs $ID
Many jobs at one time: read the stream
--------------------------------------
qex events --json
That command writes one JSON object on one line for each change of state, as it
happens. Read it in place of a loop that asks about each job. Twenty jobs give
one stream, and you learn of each result at the moment of the result.
Keep the `seq` number of the last line that you read. Give it to
`--since` when your program starts again, and you lose nothing:
qex events --json --since 348
Run `qex help events` for the lines, the numbers and the gaps.
If you operate inside a harness
-------------------------------
`qex wait` blocks. Your harness, and not qex, tells you when a background
command ends. Put the two together:
ID=$(qex submit -- make test) # gives the id at once
qex status $ID --wait # run THIS in the background of your harness
qex watches the process correctly, and your harness reports the end of the
command. You thus need no timer and no second command.
Use `qex status --wait` and not `qex wait` for this. It blocks in the same way
and it gives the same exit code, and its output also holds the state, the exit
code and the last lines of the error output. One command gives everything.
`qex submit` writes the job UUID to stdout and writes nothing else. You can
thus put the UUID in a shell variable.
A shell variable does not last between your commands. Use `--id-file` to keep
the id in a file:
qex submit --id-file build.id -- make
qex status \"$(cat build.id)\" --wait
PUT THE ID FILE WHERE IT LASTS LONGER THAN YOUR SESSION. Your project directory
or your home directory is correct. A scratch directory that your harness owns is
NOT correct, and neither is /tmp: the job continues when your session stops, but
the file goes with the session, and you then have no handle for a job that still
operates. qex gives a warning when the file goes to such a directory.
If you lose an id, `qex list` shows each job with its directory and its command,
and `qex list --cwd .` shows the jobs of this directory only.
Your session can stop, and the work continues
---------------------------------------------
THIS IS THE PROPERTY THAT MAKES qex SAFE FOR AN AGENT THAT A PERSON CAN STOP.
The job is not a child of your shell, and it is not a child of your agent. qex
starts a supervisor in its own session, and the supervisor starts the job. Three
things follow, and all three matter:
Somebody stops your agent the job continues.
Your terminal closes the job continues.
The coordinator stops or is replaced the job continues, and it still writes
its result.
Each line is true for a job of `qex run` as well, with one exception: Ctrl-C or
a SIGTERM on the waiting `qex run` stops the job. See WHAT YOU GIVE UP above.
Nothing is lost, because the record of the job is on the disk and not in the
memory of a process. Your wait is the only thing that stops.
You can therefore attach the wait again, in a later session, in a new shell,
from a different agent, at any time:
qex status $ID --wait
The id is the handle. That command gives the same answer whether the job
operates now, stopped one second ago, or stopped last night. A job that stopped
while nobody watched loses nothing at all.
This is what a monitor script cannot do. A monitor holds the answer in its own
memory: stop the monitor, and the answer is gone. Keep the id in a file with
`--id-file`, and the answer waits for you instead.
qex submit --id-file build.id -- make # session 1
# the person stops the agent here. `make` continues.
qex status \"$(cat build.id)\" --wait # session 2, and the result is there
A person can thus stop you at any moment with no cost.
Give each submission a key, and a second run starts nothing
-----------------------------------------------------------
YOU LOSE YOUR CONTEXT AND YOU RUN YOUR SCRIPT AGAIN. Without a key, qex starts a
SECOND copy of a four-hour training run beside the first copy. Both copies then
hold the machine, and both write to the same files.
Give the submission a key. The second run of the same script starts nothing:
ID=$(qex submit --dedupe-key train:$(pwd) -- uv run train.py)
qex wait $ID
The second run gives THE SAME id and exits with the code 0, so your script does
not change and `ID=$(qex submit ...)` stays correct. qex writes the reason to
stderr:
qex: this submission started no job. The dedupe key `train_home_me_p`
gives the job 7f3c8a12-..., and that job is in the state `running`.
DO NOT READ `qex list` AND DECIDE FOR YOURSELF. That test is a PROXY: you read
the list, you decide, and you submit, and a different agent can submit between
your read and your submission. The coordinator makes the test and the submission
ONE step. Two commands in the same moment thus give one job and one id.
--dedupe-key KEY start no second job while a job with this key waits or
operates. The key is free when that job stops.
--dedupe-window 1h keep the key of a job that SUCCEEDED for this time
also. A job that did not succeed never keeps its key,
because the remedy for a failure is another run.
--json write {\"id\": \"...\", \"deduplicated\": true} in place of
the id alone. Use it when your script must know if IT
started the work.
Choose a key that names the work AND the place: `build:$(pwd)`. A key such as
`build` alone stops the build of every other project on the machine.
THE WINDOW OF THE COMMAND THAT ASKS APPLIES, and not the window of the job that
holds the key. The window is a question: how old an answer do you accept? A
command that gives no window thus starts a new job, although a different command
gave a window a moment before. Give the same window in each command that shares
a key. This concerns a job that already SUCCEEDED only, so no second copy of
work that operates can start.
`qex run --dedupe-key` waits for the job that the key gives. CTRL-C THEN STOPS
YOUR WAIT ONLY, because a different agent can be the owner of that job. qex says
so when it attaches, and the wait gives the code 124: your wait stopped, and the
job continues. Use `qex kill <id>` to stop the job itself.
`qex status <id>` shows the key of a job. You can thus see which key gave you an
id, and `qex status <id>` gives the result of a job that stopped.
One command gives the result and the cause
------------------------------------------
`qex status` of a job that did not succeed also writes the last lines of its
standard error. You thus need no second command for the usual question.
qex status $ID the state, the exit code and the last lines
of BOTH streams
qex status $ID --wait the same, but wait for the job first
qex status $ID --tail 50 more lines
qex status $ID --stderr one stream only
qex status $ID --grep ERROR the lines that match
qex status $ID --no-logs the state only
qex gives both streams, because a program frequently writes its result to the
standard output and its failure summary to the standard error. The error alone
reads as a complete failure.
`qex wait` stops until the job stops. Its exit code tells you the result:
0 the job succeeded
1 the job failed
123 the job never started; it reached its `--max-queue-time`
124 your wait timed out; the job still operates
125 something stopped the job
126 the job did not run, because a job that it needed failed
127 there is no job with that id
Add `--timeout` to limit your wait. Example: `qex wait $ID --timeout 30m`.
A timeout stops your wait only. It does not stop the job.
Give up on a job that never starts
----------------------------------
A job waits until the machine has capacity for it. On a busy machine that wait
can be long, and a job with a claim that no budget can meet waits with no end.
ID=$(qex submit --max-queue-time 30m -- make test)
qex wait $ID # this gives an answer inside 30 minutes
The job does not start after that time. Its state becomes `expired`, `qex wait`
gives the code 123, and `qex status` says what the job waited for. Nothing ran,
so there is no output to read.
The clock starts at the submission. A coordinator that stops and starts again
continues the same count, so a restart does not give the job a new full wait.
qex counts the wait in whole seconds. A job can thus give up as much as one
second BEFORE its limit, and the time in the record is that count of seconds.
Give a limit of a minute or more, where one second changes nothing.
The wait for a job in `--needs` counts also. Give a value that covers the whole
pipeline, or give no value on a stage that waits for an earlier stage.
There is no value for this option by default, and there is none in the config
file until you write one. A job that qex discards is work that a person wanted,
so qex never chooses that for you.
A job takes this value at its SUBMISSION. A job that already waits in the queue
keeps the value that it had, so a change to `[defaults] max_queue_time` reaches
the jobs that you submit after it, and no earlier job.
What qex captures
-----------------
`qex submit` copies your environment and your current directory. Your job thus
operates in the same way as a command that you type now. Use `--env K=V` to add
or replace one variable. Use `--env-capture minimal` if your shell holds
secrets.
Resource claims
---------------
Give `--cpu` and `--mem`. qex uses these claims to decide how many jobs operate
together. Claims stop two agents from starting too much work at the same time.
If you do not know the size of the task, use a word in place of a number:
qex submit --cpu guess --mem guess -- ./unknown-task
half, guess one half of the budget. Two such jobs operate together.
full, max the full budget. The job operates alone.
Use `guess` to start an unknown task safely. The words also operate in a job
file:
[resources]
cpu = \"guess\"
mem = \"half\"
Do not measure a task before you run it
---------------------------------------
Do not run a small test job to find the size of a task. That method costs you
time and gives a poor measurement, because a small job does different work.
Give `--cpu guess --mem guess` and start the REAL task. That run gives you a
true measurement, and it does the work at the same time.
qex then uses that measurement for you. The next job of the same command gets a
claim from the earlier runs, so you give no claim at all:
qex submit --cpu guess --mem guess -- ./task # run 1
qex submit -- ./task # run 2: the claim is ready
`qex status` says where a claim came from.
Read the numbers yourself when you want an exact claim:
qex status $ID --json # the usage field gives max_rss and cpu_secs
The first run with `guess` is thus not wasted effort: it produces both the
result and the measurement that makes every later run cheap. What is wasted is
a separate test job that produces no result.
If your claim is larger than the full budget, qex starts the job alone when no
other job operates. The job can then swap or stop with an out-of-memory error.
The status field `forced` is `true` for such a job. That result is data: your
claim or the machine is too small.
qex learns the size of a task
-----------------------------
qex records what each job really used, and it uses those numbers as the claim
for the next job of the same command. You thus give no claim at all after the
first run:
qex submit -- cargo test # run 1: the default claim
qex submit -- cargo test # run 2: the claim comes from run 1
`qex status` says where a claim came from. The record is for the command, and
not for the name, because `cargo build` and `cargo test` need different sizes.
qex uses the LARGEST measurement that it holds, and it adds a margin. A claim
that is too small stops the job, and a claim that is a little too large costs
some capacity only.
qex records a job that completed, and a job that the kernel stopped for memory.
The two give different evidence, and qex keeps them apart:
- a job that COMPLETED gives the memory that the job needs;
- a job that the KERNEL STOPPED AT ITS OWN LIMIT gives a lower bound. The true
need is above that value, so the next claim is above it as well. A smaller
run that succeeds later does not remove that lesson. qex records this
measurement when it applied the limit itself; see `qex help config`.
qex records nothing else. A job that you stopped, or that reached its time
limit, shows the memory that it reached and not the memory that it needs.
In short: give `guess`, start the task, and read the result. Add an exact claim
later, and only if you repeat the task. After the first run of a command, qex
gives the claim for you.
A pipeline of stages
--------------------
Do not put the stages of a pipeline in one script. If stage 3 of that script
fails, you get one exit code and one log file with the output of every stage
mixed together, and you must find the cause.
Give each stage its own job, and name the jobs that must succeed first:
BUILD=$(qex submit --name build -- make)
TEST=$(qex submit --name test --needs $BUILD -- make test)
SHIP=$(qex submit --name ship --needs $TEST -- ./deploy.sh)
qex wait $SHIP
Keep the id of each stage and give the id to the next stage. An id names one
job for ever, so the script stays correct when you run it again.
Each stage has its own log file, its own exit code and its own claim. If `build`
fails, `test` and `ship` do not start. Their state becomes `skipped`, and their
record names the job that failed:
qex list
ID STATE NAME ... NOTE
a1b2c3d4 failed build ... the job stopped with the exit code 2
b2c3d4e5 skipped test ... the job a1b2c3d4 (build) is failed, ...
c3d4e5f6 skipped ship ... the job a1b2c3d4 (build) is failed, ...
There is one failure only, and it is the cause. Run `qex logs a1b2c3d4` to read
the output of that stage, and no other output.
Each skipped job names the first job that failed, and not the job before it. A
read of the last stage thus gives you the cause immediately.
--needs <id>,<id> wait for these jobs, and stop if one does not succeed
--after <id>,<id> wait for these jobs, whatever their result
Use `--after` to control the order only. A cleanup job that must run after a
build, and must run also when the build fails, uses `--after`.
`qex wait` gives the code 126 for a skipped job, and the code 1 for a job that
failed. Your script can thus separate a failure of your stage from a failure of
an earlier stage.
A job can name the jobs that you started before it. A job cannot name a job that
does not exist, so a circle of dependencies is not possible.
An id and a name have different rules
-------------------------------------
An ID must exist. That is the only rule. qex accepts an id whatever the state
of that job, so a script can submit its last stage even when the first stage
already failed. The last stage then becomes `skipped` with the correct cause.
A NAME must give a job that is in the queue or operates. A name can give a job
of an earlier run: you write `--needs test`, you forgot to start a new test job,
and the name gives the test job of yesterday. That job already succeeded, so
your stage would start immediately and wait for nothing. qex refuses a name in
that case and tells you what happened.
Use an id in a script. Use a name when you type a command yourself.
Other useful options
--------------------
--retries 3 run the job again when it fails, up to 3 times.
The job keeps one id and one record, and the log holds
every attempt. Use it for a fault outside the task,
such as a network that is not ready.
You do not need this option for a job that the kernel
stops for memory. qex raises the claim and starts such
a job again by itself, and that correction does not use
this count. See `qex help config`.
--nice N how much the job gives way to the work of a person.
-20 to 19, and a larger number gives way. The default
comes from `[politeness] nice`, and it is 10. Use
`--nice 0` to ask that this job does not give way.
qex can only make a job give way MORE than the
coordinator does: a coordinator that a user started
under `nice 5` keeps its jobs at 5 or above, because
a lower number needs privilege.
--lock NAME two jobs with one lock name never operate together.
Use it for work that shares something that a claim
cannot express: a build directory, a port, a database.
`qex run --lock target -- cargo test` stops two builds
from destroying each other in one directory.
--dedupe-key KEY start no second job while a job with this key waits or
operates. qex gives the id of that job and exits with
the code 0. Use it in a script that can run a second
time: `--dedupe-key build:$(pwd)`.
--gpu N claim N devices from the pool `gpu`. qex says WHICH
index the job gets and writes CUDA_VISIBLE_DEVICES.
--vram SIZE claim SIZE on EACH GPU that this job gets. qex does
NOT add the memory of the devices together.
--claim NAME=N claim N units of the pool NAME. `--lock NAME` is the
same as `--claim NAME=1`.
Run `qex help resources` for the pools.
--id-file FILE write the job id to a file as well as to stdout.
qex wait A B --any give control back when the FIRST job stops.
qex rerun <id> submit the same job again, with a new id.
If the coordinator is older than your command
---------------------------------------------
A coordinator operates for hours, and a new build can replace the qex program.
The coordinator then holds earlier code.
qex asks the coordinator what it can do, and it REFUSES a job that the
coordinator cannot obey:
qex: the coordinator (pid 3507877) is version 0.3.0, and it cannot
obey --lock.
qex refuses this job. The coordinator would ignore that option in
silence, give you a job id, and run the job without the rule that you
asked for.
A refusal is safer than a job that starts. A job specification travels as JSON,
and a field that the coordinator does not know is ignored with no message. A
lock that nothing applies looks exactly like a lock that operates, until two
jobs destroy each other.
The coordinator stops when no job operates, and the next command starts one that
can obey. `kill <pid>` changes it at once; the jobs that operate continue,
because a new coordinator reads the same records.
`qex version` gives what your command can do and what the coordinator can do.
Other commands
--------------
qex list --json all the jobs and their states
qex status <id> --json one job in detail, with the last error lines
qex logs <id> --follow the output while the job operates
qex kill <id> stop a job that operates
qex cancel <id> remove a job from the queue
qex clean --state done delete the records of the jobs that stopped
qex info the coordinator and the free capacity
Every command that reads data accepts `--json`. Use `qex schema status`,
`qex schema job` and `qex schema pipeline` to get the JSON Schema of each
format.
Short forms of a job id
-----------------------
Each command accepts the first characters of an id, and it also accepts the
name of a job:
qex submit --name build -- make
qex status build
Give a name with `--name` when you start a job. You can then use that name in
place of the id, and you do not keep the UUID.
A name that gives more than one job gives an error with the list of the jobs.
Start with a job file for a complex task
----------------------------------------
qex submit --job train.toml
Run `qex help job-file` for the fields.
For several stages in one file, use a pipeline:
qex pipeline ci.toml
Run `qex help pipeline`. A pipeline gives each stage a name that belongs to
that one submission, so two runs of one file never share a name.
One command and many inputs
---------------------------
GROUP=$(qex submit --each-line inputs.txt -- ./process {})
qex list --group $GROUP
One job for each line of the file, and one group id for all of them. `{}` takes
the text of the line. qex starts no shell, so a line becomes exactly one
argument and never a command. Run `qex help each-line`.
Tell the people who make qex
----------------------------
qex exists because somebody measured their own monitor scripts and gave the
numbers. Each of the parts above came from a report of that kind.
Something helped you:
https://github.com/stephenc/qex/issues/1
Leave a comment. Give the number if you have one: the hours, the jobs,
the size of the machine. Say which model you are and which harness runs
you, because the faults are not the same in each harness.
A recipe that is not in the documentation:
https://github.com/stephenc/qex/issues/2
qex has no timer, because cron and `qex submit --lock` do it better.
That recipe is there, with the guard that stops the queue from growing.
Something did NOT work, or a word sent you the wrong way:
LOOK FOR IT FIRST, AND ADD TO WHAT IS THERE.
gh issue list --repo stephenc/qex --state all --search \"your words\"
A second report of one fault says that the fault is common, and that is
useful. A second ISSUE for one fault divides the answer between two
pages, and the person who corrects it must read both and close one.
Add your machine, your model and your harness to the issue that exists:
gh issue comment <number> --repo stephenc/qex
Open a new one only when nothing matches:
https://github.com/stephenc/qex/issues/new/choose
A SMALL FAULT IS WORTH AN ISSUE. An option that you looked for and did
not find, a message that made you look in the wrong place, an exit code
that surprised you: those reports are the ones that make a tool good.
Give four things, and no more: the command in full, what qex wrote in full
(use --json), what you expected, and the output of `qex version`.
DO NOT PUT THE OUTPUT OF `--show-env` IN A PUBLIC PAGE, and do not put anything
that a token can hide in. `qex submit` copies the environment of your shell, so
that output is a secret. This is the same rule as `qex help output`.
If you cannot reach GitHub, give the words to the person that you work with and
let them decide.
";
pub const JOB_FILE: &str = "\
qex job files
=============
A job file describes one job. Use a job file for a long command, for many
environment variables, or to keep the job in your repository.
qex submit --job train.toml
qex reads TOML, YAML and JSON. The file extension selects the format. TOML is
the format in this documentation.
One job file holds ONE job. For several stages in one file, use a pipeline file
and the command `qex pipeline`. Run `qex help pipeline`.
A minimal file
--------------
command = [\"uv\", \"run\", \"train.py\"]
A full file
-----------
name = \"train-model\" # the name in `qex list`
cwd = \"/home/me/project\" # the default is your current directory
command = [\"uv\", \"run\", \"train.py\", \"--epochs\", \"50\"]
timeout = \"4h\" # the default is no limit
max_queue_time = \"30m\" # give up if the job waits this long
tags = [\"ml\"] # for `qex list --tag ml`
priority = 0 # a larger number starts earlier
needs = [\"build\"] # stop if these jobs do not succeed
after = [\"cleanup\"] # wait for these jobs, whatever the result
env_capture = \"all\" # all, minimal or none
nice = 10 # -20 to 19; a larger number gives way
no_limit_env_hints = false # true: do not tell the job its claim size
dedupe_key = \"train:p1\" # start no second job while this one operates
dedupe_window = \"0\" # keep the key after a job that succeeded
[resources]
cpu = 3
mem = \"8GB\"
gpu = 2 # devices from the pool `gpu`
vram = \"20GB\" # on EACH device that this job gets
[resources.claims]
net = 1 # 1 unit of the pool `net`
[env]
HF_HOME = \"/data/hf\"
Fields
------
`command` is a list of arguments. It is not a shell command line. qex does not
start a shell, so you need no quotation marks and no escape characters. To use
a shell feature such as a pipe, name the shell:
command = [\"bash\", \"-lc\", \"a | b > c.txt\"]
`mem` accepts `8GB`, `8G`, `512MB` or a number of bytes. One unit step is 1024.
`gpu` and `vram` claim the pool `gpu`. `vram` is the quantity on EACH device
that the job gets, and qex does NOT add the memory of the devices together.
With no `vram`, the job takes the whole of each device that it gets.
`[resources.claims]` claims the other pools. Give a number, or a table with a
count and a size: `tpu = { count = 2, size = \"8GB\" }`.
DO NOT SET `CUDA_VISIBLE_DEVICES` IN `[env]` FOR A JOB THAT CLAIMS A GPU. qex
gives the devices to the job and writes that variable, so the two values would
disagree. qex refuses such a job and says so.
`timeout` accepts `30s`, `5m`, `4h`, `2d`, or `0` for no limit.
`max_queue_time` accepts the same values. It limits the time that the job WAITS,
and `timeout` limits the time that the job RUNS. A job that reaches this limit
does not start, and its state becomes `expired`. The time counts from the
submission, and the wait for a job in `needs` counts also.
`env_capture` selects the environment that the job receives:
all every variable from your shell (the default)
minimal PATH, HOME, USER, LOGNAME, SHELL, LANG, TZ only
none no variable from your shell
`dedupe_key` makes the submission idempotent. While a job with that key waits or
operates, a second submission with the same key starts NO job: qex gives the id
of the first job and exits with the code 0. Use it for a job file that a script
submits each time it runs. `--dedupe-key` on the command line replaces the value
in the file.
`dedupe_window` accepts a time such as `1h`. The key of a job that SUCCEEDED
stays for that time. A job that did not succeed never keeps its key, because the
remedy for a failure is another run. The default is `0`.
A pipeline stage has NO dedupe key. A key on one stage would answer for that
stage alone, and the stages after it would wait for a job of an earlier run.
The sequence of the sources
---------------------------
A later source replaces an earlier source:
environment from the shell -> job file [env] -> --env K=V
directory from the shell -> job file cwd -> --cwd D
config file defaults -> job file -> command line options
Secrets
-------
qex writes your captured environment to `spec.json` with mode 0600. If your
shell holds secrets, use `--env-capture minimal`. The command `qex status` hides
the environment. Add `--show-env` to see it.
A field name with a spelling error gives an error. qex does not ignore it.
";
pub const PAUSE: &str = "\
qex pause and qex resume
========================
Use these commands to take the machine, or one resource of it, back for a
moment.
qex pause queue start no new job
qex resume queue start the queue again
qex pause lock <name> take that lock for yourself
qex resume lock <name> give the lock back
qex pause say what is paused now
`qex resume` with no word starts the queue again.
Pause the queue
---------------
qex pause queue --reason \"recording a demo\"
qex then starts NO job. There is no exception: every job in qex has a claim, so
a job that costs nothing does not exist, and `paused` is one fact that you can
act on.
THE JOBS THAT OPERATE NOW CONTINUE. Each one already holds its capacity, and a
stop would lose that work. To wait for a quiet machine:
qex pause queue --drain # gives control back when no job operates
To stop a job that operates, use `qex kill <id>`.
Give the pause an end
---------------------
qex pause queue --for 30m
A pause with no end continues until you run `qex resume`. Every command that
lists jobs says so, because a pause that a person forgets gives an empty queue
in the morning.
A second `qex pause queue` KEEPS the end and the reason of the first one. A
command that replaced them would change a pause of 30 minutes into a pause with
no end. To replace an end, run `qex resume queue` first.
`--for 0` is an error. To end a pause now, run `qex resume queue`.
A job with `--retries` starts its next attempt inside its own supervisor, and
that supervisor reads the pause as well. The next attempt waits.
Pause a lock
------------
A lock names a resource that one job at a time may hold (`qex submit --lock
gpu0`). You frequently need that same resource by hand.
qex pause lock gpu0
qex gives the lock TO YOU as soon as no job holds it. Every job that needs it
waits, and `qex list` gives the reason:
b0bb2614 queued train ... waits for the lock `gpu0`, which a person holds
The command never fails when a job holds the lock now. qex records the request,
that job keeps the lock, no other job takes it, and the lock comes to you when
that job stops. The command is thus safe to type at any moment.
What survives
-------------
The pause is a file beside the job records, so it survives a coordinator that
stops. A new coordinator reads it and the queue stays paused.
If qex cannot read that file, it HOLDS the queue and says so. A file that qex
cannot read can hold a pause, and qex does not know. `qex resume queue` writes
a new file and starts the queue again.
The pause covers YOUR queue only. It does not pause another user of the
machine. `qex info` says so.
What a pause does NOT do
------------------------
A pause does not expire a job. `--max-queue-time` measures the time that a job
waits for the QUEUE, and a person who holds the machine is not the queue. The
clock of that limit stops at the pause and runs again at the resume, so a pause
of 30 minutes does not kill every job with a smaller limit. `qex status` gives
that time in `queue_pause_secs`.
This holds when the pause ends by itself while no coordinator operates. The next
coordinator finds it and gives the time back.
A pause of a LOCK does not stop that clock. A job that waits for a lock already
expires in the same way, whatever holds it. Give such a job a
`--max-queue-time` that covers the hold, or no limit at all.
A pause refuses no command. `qex submit` gives you a job id and the exit code 0,
and the job waits with the pause as its reason.
Who may end it
--------------
Anybody who can reach this queue. A pause is not a lock on the queue: the queue
belongs to one user of the machine, and everybody who reaches it already shares
every job in it. Each line below names the pid that asked for the pause, so you
can find the owner before you start the queue again.
Where to read it
----------------
qex pause what is paused, for how long, and who asked
qex info the same line, with the budget and the load
qex top the same line, on the page
qex list the same line, before the jobs
qex wait <id> the same line, before the wait begins
qex status <id> the reason that one job waits
";
pub const CONFIG: &str = "\
qex configuration
=================
The config file is `~/.config/qex.toml`. The file is optional. Run
`qex config path` to see its location and `qex config show` to see the values
that qex uses now.
[budget]
cpu = \"75%\" # cores that qex can use; an integer or a percentage
mem = \"75%\" # memory that qex can use; a size or a percentage
[system]
reserve_mem = \"2GB\" # memory to keep free for other programs
max_pressure = 20 # maximum PSI memory pressure (Linux only)
[enforce]
mode = \"off\" # off, soft or hard
mem_overcommit = 1.5 # soft mode: memory.max = claim * this value
use_systemd = true # permit a temporary systemd unit for the cgroup
[peers]
enabled = true
dir = \"/tmp/qex\"
stale_after = \"30s\"
[queue]
oversized = \"run-when-idle\" # run-when-idle, reject or queue
settle = \"3s\"
max_bypass = 2 # jobs that may start before the job at the front
[politeness]
nice = 10 # -20 to 19; a larger number gives way
io = \"none\" # none, best-effort or idle (Linux)
oom_score_adj = 0 # a larger number offers the job to the OOM killer
# first (Linux)
[submit]
env_capture = \"all\" # all, minimal or none
minimal_env = [\"PATH\", \"HOME\", \"USER\", \"LOGNAME\", \"SHELL\", \"LANG\", \"TZ\"]
[claims]
export_env = true # tell the job how large its claim is
also = [] # \"java\", \"make\", or both
[learn]
enabled = true # use the earlier jobs of a command as the claim
margin = 1.5 # the multiplier for a measurement
[logs]
max_bytes = \"32MB\" # the output that qex keeps for each stream of a job
[retry]
on_oom = 2 # times to raise the claim after a kill for memory
growth = 2.0 # the multiplier for the claim at each raise
[history]
keep = \"1d\" # how long to keep the id of a job after its removal
[gc]
keep = \"1d\" # the age of a record that `qex gc` deletes
[defaults]
cpu = 1 # the default is 1 core
mem = \"2GB\" # the default is the machine memory / the core count
timeout = \"0\" # the default is no limit
max_queue_time = \"0\" # the default is no limit on the wait
vram = \"0\" # 0 means: a job with no --vram takes the whole device
# A pool with devices. qex says WHICH one each job gets.
[[pool]]
name = \"gpu\"
size = \"vram\" # the quantity each device holds
devices = [\"24GB\", \"24GB\", \"24GB\", \"24GB\"]
env = \"CUDA_VISIBLE_DEVICES\"
# A pool with no devices. The number is sufficient.
[[pool]]
name = \"net\"
count = 4
[hooks]
on_stop = [] # the command that qex runs when a job stops
on_stop_states = [\"completed\", \"failed\", \"killed\", \"timeout\", \"expired\", \"oom\"]
timeout = \"30s\" # the time limit for that command
Quotation marks around a number
-------------------------------
A field that takes a number, a size, a time or a percentage accepts the value
with quotation marks and without them. `cpu = 2` and `cpu = \"2\"` give the same
budget, and `margin = 1.5` and `margin = \"1.5\"` give the same margin. A size
with no unit is bytes, and a time with no unit is seconds.
The quotation marks do not change WHICH values a field takes. `[budget] cpu`
takes a percentage, because it gives a part of the machine to all the jobs
together. `[defaults] cpu` gives the cores for ONE job, so it takes a whole
number only, and a percentage there gives an error.
A command when a job stops
--------------------------
`[hooks] on_stop` names a command that qex runs each time a job reaches its
final state. Use it for a notification: a message on the screen, a line in a
file, or a message to a chat. A person who left the machine thus learns that
the job of four hours stopped.
[hooks]
on_stop = [\"notify-send\", \"a qex job stopped\"]
The hook is in the config file only. A job file has no hook field. The hook
belongs to the machine and to the person at it, and not to the work: the same
pipeline runs on a laptop with a screen and on a build machine with none.
Name an ABSOLUTE path. The hook starts in the directory of the job, and the
person who submitted the job chose that directory, so `[\"./notify\"]` selects a
program that the submitter can put there.
The value is a program and its arguments. qex starts no shell, in the same way
as for a job. To use a shell feature, name the shell:
[hooks]
on_stop = [\"bash\", \"-lc\", \"echo \\\"$QEX_JOB_NAME $QEX_STATE\\\" >> ~/qex.log\"]
The job supplies these variables. A variable with no value is empty text.
QEX_JOB_ID the job id
QEX_JOB_NAME the job name, in the safe form that `qex list` shows
QEX_STATE the final state
QEX_EXIT_CODE the exit code of the job, if the job ran to its own end
QEX_SIGNAL the signal number, if a signal stopped the job
QEX_ELAPSED_SECS the seconds that the job ran
QEX_CWD the directory of the job
QEX_JOB_DIR the directory of the record, which holds the logs
QEX_ATTEMPTS the number of times that qex started the job
QEX_MAX_RSS the maximum memory in bytes
QEX_TAGS the tags, separated by a space
The values arrive in the environment and never in a command line. qex builds no
text that a shell reads, so a job name such as `x; rm -rf ~` is a name and never
a command, whatever the hook does with it.
QEX_JOB_NAME is the SAFE name: the letters, the numbers and `-_.` only, which is
the one form of a name that qex shows anywhere. A hook puts a name in front of a
person, and a raw name with an ESC byte in it moves the cursor of a terminal and
writes over the text around it. That name goes back into `qex status` as it
stands. A hook that needs the name that the submitter typed reads `status.json`
in QEX_JOB_DIR. QEX_TAGS and QEX_CWD have no such rule, so qex replaces each
control character in them with a space.
QEX_EXIT_CODE is the code of the JOB. It is empty for a job that something
stopped, because such a job gave no code of its own. Read QEX_STATE first: it
holds the same word that `qex status` prints. Run `qex help exit-codes` for the
codes of the commands.
`on_stop_states` selects the jobs that give a message. The default list holds
each state of a job that ran, and `expired`: a job that gave up waiting never
ran, so nothing else says so. `cancelled` and `skipped` are not in it: you
cancelled the job yourself, and one failure in a pipeline of twenty stages
would give twenty messages. Add those names to get them. For a message on a
failure only:
[hooks]
on_stop_states = [\"failed\", \"timeout\", \"oom\"]
The hook cannot damage the queue. qex runs it after the final state is on the
disk, so the job has its result, the budget is free, and the next job starts
before the hook does anything.
qex never runs the hook two times for one job. It runs the hook one time for
each job that stops, EXCEPT when the machine or the process stops in the moment
between the record of the run and the run itself: qex then loses that message,
and it does not try again. A message that arrives two times is worse than a
message that is lost.
A hook that uses more than `[hooks] timeout` receives TERM and then KILL, in a
process group of its own. The hook writes into a pipe and never into a file, so
at 1MB of output qex stops reading, shuts the pipe and stops the hook. The two
streams of the hook thus stop growing AT 1MB. This bounds those streams only: a
hook that opens a file of its own is a program that you chose to run. That size is fixed, and it is NOT `[logs]
max_bytes`, which limits the output of a job. A hook that
fails does not change the job, and qex writes nothing in the `error` field of
the job for it.
A hook is not a job. It does not take the `[politeness]` values, because those
make work give way to a person and a notification is FOR the person. It does not
join the cgroup of the job, so `[enforce]` puts no memory limit on it. It
receives no variable of `[claims]`, because it makes no claim on the budget.
qex reads the config file at each job that stops. A hook that you delete thus
runs no more, and a hook that you add runs at once. You do not restart the
coordinator.
The output goes to `hook.log` in the directory of the job. Read it with
`qex logs <id> --hook`, which also gives the verdict of qex: a hook that did not
start, a hook that was too slow, or a hook that stopped with an error.
Pools
-----
A pool is a name and a total. A job claims units of a pool with `--claim
NAME=N`, and `--lock NAME` is the same as `--claim NAME=1`.
A pool with `devices` also gives a capacity to each device. qex then says WHICH
device each job gets, writes the index into `QEX_<NAME>_DEVICES`, and writes it
into the variable that `env` names.
Give `count` or `devices`, and not both. Use `devices` when qex must say WHICH
one a job gets. Use `count` when the number is sufficient.
A pool cannot use the name `cpu` or `mem`. Those two are in `[budget]`.
A pool that this file declares is shared with the other users of the machine:
each coordinator publishes the units and the device indices that it gave away.
A name that this file does NOT declare is a lock, and a lock stays inside one
queue.
qex reads no driver. The devices come from this file only, so a machine with no
CUDA and no driver library schedules GPU claims correctly.
Default job size
----------------
A submission without `--cpu` or `--mem` uses the `[defaults]` section. If that
section gives no value, qex uses 1 core and an equal part of the machine
memory. On a machine with 16 cores and 32GB, the default job is 1 core and 2GB.
The default job size thus scales with the machine.
The limit on the output of a job
--------------------------------
`[logs] max_bytes` is the space that one stream of one job can use. The default
is 32MB for `stdout.log` and 32MB for `stderr.log`.
qex applies this limit WHILE THE JOB WRITES. A job that writes 400MB thus never
puts 400MB on the disk. The same disk holds the record of each job, and qex is
made to be started and left, so a job with no limit can fill that disk while
nobody looks.
qex keeps the first part of the output and the last part. The first part holds
the start-up and the configuration. The last part holds the failure. Between
the two, qex writes a line that says how many bytes and how many lines went:
[qex] ---- 361MB and 4201177 line(s) of the output are not in this file ----
`qex status` and `qex logs` also give that count, so a reader always knows that
the file is not the whole output. `qex status --json` gives it in the field
`logs_dropped`.
Use `max_bytes = \"0\"` for no limit. The words \"none\", \"never\" and \"unlimited\"
do the same, and they are the words that `[defaults] timeout` takes. Then a job
can fill the disk.
The supervisor of a job reads this field one time, when the job starts. A change
to the file thus does nothing to a job that already writes, and it controls the
next job to start. The supervisor reads the file itself, so the new value does
not wait for the coordinator to read the file again. To give a new limit to a
job that operates, stop it and use `qex rerun`.
The claim in the job
--------------------
A claim controls the queue. It does not control the job: a job that asks the
machine how many cores it has receives the number of the MACHINE. qex therefore
writes the claim into the environment of the job, and most runtimes read those
variables in place of the machine.
QEX_CPU, QEX_MEM, QEX_MEM_MB your own script: make -j\"$QEX_CPU\"
GOMAXPROCS, GOMEMLIMIT Go
OMP_NUM_THREADS OpenMP: C, C++ and Fortran
OPENBLAS_NUM_THREADS, MKL_NUM_THREADS numpy, pandas and the libraries
NUMEXPR_NUM_THREADS below them
VECLIB_MAXIMUM_THREADS Accelerate, on macOS
RAYON_NUM_THREADS, CARGO_BUILD_JOBS Rust
JULIA_NUM_THREADS Julia
DOTNET_PROCESSOR_COUNT .NET
POLARS_MAX_THREADS Polars
NODE_OPTIONS node, at 3/4 of the claim
qex writes these ONLY when you gave both `--cpu` and `--mem`. A default claim
and a learned claim are not a decision that you made, and a job that heard
`one core` would run single-threaded. qex never replaces a value that is
already there.
`[claims] also` adds two more. Each of the two has a cost, so neither is a
default:
java JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=N -XmxMm
Every JVM then writes `Picked up JAVA_TOOL_OPTIONS: ...` to its
standard error, and that line goes into the log of the job.
make MAKEFLAGS=-jN
A Makefile that gives its own `-j` wins, so this changes a Makefile
that gives none. It thus makes a build parallel that its author
never ran in parallel, and a Makefile with an incomplete dependency
graph then fails.
Turn it all off with `export_env = false`, or for one job with
`qex submit --no-limit-env-hints`.
A kill for memory
-----------------
The kernel stops a job that uses more memory than its claim. That kill says one
thing: the claim was too small. qex raises the claim and starts the job again,
up to `[retry] on_oom` times, and it multiplies the claim by `growth` at each
raise. The default values give 2 raises and 4 times the first claim.
This count is separate from `--retries`. `--retries` is for a fault outside the
task, and you chose that number for that fault. The claim is usually the work of
qex, so qex corrects its own fault and does not spend your count.
The claim never goes above `[budget] mem`. A job that already claims the whole
budget keeps the state `oom`, and the record says that you need a larger machine
or a larger budget. The job also goes through the QUEUE again, because the queue
never admitted the new claim.
A new attempt is the SAME job: one id, one record and one log file. The job
hears the raised claim in `QEX_MEM` and in the other claim values, the log holds
each attempt behind a line `--- attempt 2 ---`, and `qex status` gives the count
in `attempts`. `--max-queue-time` expires no new attempt, because that value
limits the WAIT of a job and this job already ran. The stop hook runs one time,
for the state that the job STOPS in.
qex acts on the evidence of THIS JOB only. With `mode = \"soft\"` or
`mode = \"hard\"` above, qex makes a cgroup for each job and reads the count of
that cgroup: the kernel stopped the job at the claim, so the claim was too
small. With `mode = \"off\"`, which is the default, qex reads the count of your
login session, and that count also rises when the kernel stops a DIFFERENT
program of the same user. qex then reports the state `oom` and starts no new
attempt: the machine can be full while the claim of this job is correct.
Set `[enforce] mode` to get the correction. Set `on_oom = 0` to stop it.
The order of the queue
----------------------
The order is a RESERVATION WITH A BOUNDED BYPASS, and it is not strict order.
`[queue] max_bypass` gives the number of jobs that may start before the job at
the front of the queue. The default is 2. After that number, qex keeps the
capacity for the job at the front and starts nothing else, so a stream of small
jobs cannot hold a large job in the queue for ever.
With `max_bypass = 0`, no job passes a job at the front THAT KEEPS CAPACITY.
The order is then strict for those jobs, and one large job stops the queue while
it waits. The value changes nothing for the classes below that keep no capacity
at any value: qex starts the jobs behind those, because it does not control the
holder and no wait for it would end.
qex keeps capacity only while the jobs of THIS queue hold it, or while a large
job waits for a quiet machine. qex schedules those releases. If another user or
a program outside qex holds the capacity, qex starts the jobs behind the job at
the front and keeps no capacity: an empty machine gives that job nothing,
because qex does not control the holder. The count of the jobs that passed is
NOT reset when the holder changes, so the job becomes unpassable in the same
scheduler cycle in which the other user releases the capacity.
A job that waits for a job that it needs, for a lock, or for a pause never
keeps capacity. None of the three takes capacity, so there is nothing to keep,
and a paused queue starts no job at all.
A bypass does not change the queue. The order stays the order of `--priority`
and then of the submission, and the job at the front starts before every job
behind it as soon as it can start.
Enforcement
-----------
The default mode is `off`. A claim then controls the queue only, and qex sets
no limit on the job. This behaviour is the same on Linux and on macOS.
The modes `soft` and `hard` need cgroup v2, so they operate on Linux only. In
`soft` mode the kernel slows a job at its claim. In `hard` mode the kernel stops
a job at its claim. If qex cannot set a limit, it writes a warning and continues
in the `off` mode.
A key name with a spelling error gives an error. qex does not ignore it.
When the coordinator reads this file
------------------------------------
The coordinator reads this file at its start, and it reads it again when the
content of the file changes. qex looks at the file about ten times in half a
second, and it takes the content when every look gave the same content, so the
new values arrive in about half a second to one second. They apply to the jobs
that START after the change. A job that operates keeps the claim that it made.
Those looks are not a delay for its own sake. A program that writes this file
one line at a time leaves a file that stops in the middle, and a file that stops
in the middle is still valid TOML. Every key that the writer did not reach yet
takes its DEFAULT value, and a stop in the middle of a line gives a wrong value
that is not a default value: a file that is becoming `cpu = 16` reads as
`cpu = 1`. qex says nothing in either case, because it CAN read such a file.
A file that changes back and forth in step with those looks can still be taken.
qex LOOKS at the file, and it gets no message when the file changes, so a writer
that puts two whole files at the path in turn at the period of the looks gives
every look the same content. No number of looks removes that. Write this file in
one step to be safe: write a temporary file, then rename it over this one.
A file that qex cannot read does not become the default values. The coordinator
keeps the values that it had, and `qex info` says so. That covers a file with a
fault, an empty file, a file that is gone, and a path that is not a regular
file. Correct the file, and the coordinator reads it again with no other step.
The path must be a regular file, or a link to one. The open of a FIFO waits for
a writer, and a read of a device gives bytes with no end, so every command that
reads this file refuses a path of another kind and says so at once.
A NEW option is different. The coordinator holds the code that started it, so a
coordinator of an earlier build does not know a name that a later build added.
It refuses the file and keeps the values that it had. Install the new qex FIRST,
then run `qex info` for the pid and `kill <pid>` to replace the coordinator, and
put the new option in the file last.
";
pub const RESOURCES: &str = "\
qex resources and the budget
============================
Claims
------
Each job has a claim: a number of cores and a quantity of memory. Give the claim
with `--cpu` and `--mem`, or in the `[resources]` section of a job file.
A claim is an estimate of the peak use. qex uses the claims to decide how many
jobs operate together. Two agents on one machine thus do not start too much work
at the same time.
Words in place of a number
--------------------------
half, guess one half of the budget
full, max the full budget
qex calculates these words against the budget at the time of the submission, so
the record of the job holds an exact value.
Use `guess` for a task of an unknown size. Two jobs with the claim `guess`
operate together, and a third job waits. Use `full` for a task that must have
the machine to itself; every other job then waits for it.
If you give no claim, qex uses the `[defaults]` section of the config file. If
that section gives no value, a job gets 1 core and the machine memory divided by
the number of cores.
By default a claim sets no limit on the job. See `qex help config` to make qex
apply the claim as a limit.
Pools: GPUs, VRAM and counted locks
-----------------------------------
The cores and the memory are two quantities. Everything else that a machine can
count is a POOL: a name, a total, and, when qex must say WHICH one, a list of
devices.
--gpu N claim N devices from the pool `gpu`.
--vram SIZE claim SIZE on EACH GPU that this job gets.
--claim NAME=N claim N units of the pool NAME.
--lock NAME the same as `--claim NAME=1`.
qex submit --cpu 4 --mem 16GB --gpu 1 --vram 20GB -- uv run train.py
qex submit --cpu 8 --mem 32GB --gpu 2 -- uv run train.py # 2 whole devices
qex submit --claim net=1 -- ./download.sh
qex run --lock target -- cargo test
Declare a pool in `~/.config/qex.toml`. Run `qex help config` for the form.
qex does not add the VRAM of the devices together
-------------------------------------------------
A job that needs 40GB on one device cannot run on two devices of 24GB. qex
refuses such a job and says that it can never start.
`--vram SIZE` is the quantity on EACH device that the job gets. With no
`--vram`, the job takes the whole of each device that it gets. That is the safe
default: a claim that consumed nothing would let qex put four unlimited jobs on
one card. `[defaults] vram` lets you change it.
qex says which device, and it tells the job
-------------------------------------------
qex gives the devices with the most free capacity first, and the lowest index
for a tie. The job then sees both:
CUDA_VISIBLE_DEVICES=2,3 # because the pool `gpu` names this variable
QEX_GPU_DEVICES=2,3 # always, for every indexed pool
QEX_GPU_VRAM=21474836480 # the quantity on each device, in bytes
QEX_CLAIM_NET=1 # for a pool with no devices
qex status <id> # the line `devices: gpu 2,3`
qex status <id> --json # the field `assigned`
The variable is what a framework reads with no change to its code. The record is
what you read AFTERWARDS to explain a failure: the record stays, and the
environment goes with the job.
Do not set `CUDA_VISIBLE_DEVICES` yourself for a job that claims a GPU. qex
refuses that job, because the two values would disagree.
qex does not read a driver
--------------------------
The devices come from the configuration only. A machine with no CUDA and no
driver library thus schedules GPU claims correctly: a count in the file, a claim
on the job, and the same arithmetic that admits a job today.
Two users who give different device counts disagree, in the same way and for the
same reason that they can disagree about `[budget]`. The accounting is
cooperative.
A claim above the pool total is always refused
----------------------------------------------
This behaviour is different from the cores and the memory. A memory job that is
too large can run alone and swap, and that result is data. An empty machine does
not make a fifth GPU, so `qex submit --gpu 8` against a pool of 4 gives an error
at the submission, whatever `[queue] oversized` says.
When does a job start
---------------------
qex starts a job when all these conditions are true:
1. The claims of the jobs that operate, plus this claim, are in the budget.
2. The claims of the other users leave sufficient capacity.
3. Each pool that the job claims has free units, or free devices with
sufficient capacity.
4. The free memory stays above `reserve_mem` and the memory pressure is below
`max_pressure`.
If a job waits, `qex status` gives the reason in the `blocked_reason` field.
Why a job waits, and what holds the queue
-----------------------------------------
`blocked_reason` names the holder of the capacity. There are four holders, and
they do not have the same effect on the jobs behind:
1. The jobs of this queue. qex knows that they stop, so it keeps the capacity
for the job at the front after `[queue] max_bypass` jobs passed it.
2. Another user. qex does not control that user, so the wait has no known end.
qex starts the jobs behind, and it keeps no capacity.
3. A program outside qex, or memory pressure. The same rule as 2.
4. The size of the job. A job that is larger than the budget waits for a quiet
machine, or the config file keeps it in the queue.
Each job that waits gives a reason of ITS OWN. A job behind a job that qex keeps
capacity for gives that fact and the id of the job at the front.
qex counts the jobs that pass the job at the front in the field `passed_by`, and
`blocked_since` gives the time when that job reached the front. The count is not
reset when the holder changes. A job that another user held for an hour keeps its
count, and it is unpassable in the same cycle in which a job of this queue
becomes the holder.
Is the queue healthy
--------------------
qex info
The last line answers the question. The queue is healthy when a job started
recently, OR when the line names a cause outside this queue: another user or the
machine. The queue is stuck when no job started and the cause is a job of this
queue. `qex top` gives the same line in its header.
A job that is larger than the budget
------------------------------------
A claim can be larger than the full budget. Such a job can never meet condition
1, so qex starts it alone when no other job operates.
The job can then cause swap operations, use all the cores, or stop with an
out-of-memory error. Each of these results is data for you. A job that waits for
ever gives no data.
The status field `forced` is `true` for such a job, and `forced_reason` gives
the text. `qex submit` also writes a warning to stderr immediately. The UUID
stays alone on stdout.
To change this behaviour, set `[queue] oversized` to `reject` or to `queue`.
When to look at the measured use
--------------------------------
qex measures each job and writes the values in the status. You do not need a
test job, and you do not need to read the values after each job.
Give `guess` and start the real task. Look at the measured use only when both of
these conditions are true:
1. You run the same kind of task many times.
2. The jobs wait in the queue, or a job stopped with an out-of-memory error.
qex status <id> --json
The `usage` field gives `max_rss` in bytes and `cpu_secs`. A task that always
uses much less than its claim wastes capacity: put an exact claim in a job file,
and more jobs then operate together. A task that the kernel stops for memory
needs a larger claim, and qex gives it that claim itself: it multiplies the
claim and starts the job again.
For one task, this step is not necessary.
Other users
-----------
Each qex coordinator writes its current claims to `/tmp/qex`. A coordinator
reads the files of the other users before it starts a job. This method needs no
administrator rights.
This method is cooperative. A different user can write an incorrect value. qex
also tests the free memory of the machine, so it finds a load that no
coordinator reports.
";
pub const STATES: &str = "\
qex job states
==============
queued qex accepted the job. It waits for capacity.
starting qex started the supervisor. The job process starts.
running the job operates.
completed the job stopped with the exit code 0.
failed the job stopped with an exit code that is not 0.
killed the command `qex kill` stopped the job.
timeout the job used more time than its `--timeout` value.
expired the job waited more time than its `--max-queue-time` value,
so it never started. There is no output and no exit code.
oom the kernel stopped the job, because the job used more
memory than its claim.
cancelled qex removed the job from the queue before it started.
skipped a job that this job needed did not succeed, so this job
did not start. The field `caused_by` names the job that
failed first.
The states `queued`, `starting` and `running` are not final. Each other state is
final and does not change.
The state `oom` is different from `failed`. It says that the kernel stopped the
job for memory.
When qex applied the memory limit itself, that kill proves that the claim was
too small: qex raises the claim and starts the job again, up to 2 times. With no
limit, which is the default, qex reads the count of the login session, and that
count also rises for a different program of the same user. qex then reports the
state and starts no new attempt. The record of the job says which of the two
happened, and what you can do.
The state `killed` also covers a kill that qex cannot explain. The kernel and
`qex kill` both use the signal KILL, and a machine with no cgroup keeps no count
of the kills for memory. qex then gives `killed`, which starts no new attempt,
and the record says that qex could not tell.
The state `expired` is different from `timeout`. For `timeout`, the work is too
slow, and the log file holds the output of the part that ran. For `expired`, the
machine never gave the job a place, so the log file is empty. Read the `error`
field: it says what the job waited for and how long it waited.
Use `qex list --state running` to select the jobs in one state.
";
pub const EVENTS: &str = "\
qex events: one stream for every job
====================================
qex events --json
The command writes ONE JSON OBJECT ON ONE LINE for each change, as it happens.
Use it in place of a loop that asks about each job. An agent that drives twenty
jobs reads one stream, and it does not send twenty commands again and again.
qex events --json | while read -r line; do ... done
The command needs no timer. It writes each line at the moment of the change,
and it uses no CPU time while it waits.
The lines
---------
Each line has the field `event`, which gives its type. Ignore a type that you do
not know: a later version of qex can add one.
stream the first line. It gives `stream_id`, which is the name of this
stream, and the numbers that the coordinator holds. KEEP THE
NAME. See `--since`.
job the record of one job changed. See below.
gap you lost events, and this line counts them.
bye the coordinator stops now, and it says why.
A `job` line holds:
seq the number of this event. KEEP IT. See `--since`.
time the time of the change
id, name the job
state the state now
previous the state before, or null for the first line of a job
change `state`, or `reason` for a job that waits in the queue
job the whole record, the same as `qex status --json`
The field `job` holds everything, so you need no second command to learn the
exit code, the measured use or the cause of a failure.
A job that waits gives a line with `change` = `reason`. The field
`job.blocked_reason` then says what the job waits for: memory, a lock, or a
different job. The reason arrives a moment after the job enters the queue,
because the scheduler writes it.
The stream reports what the coordinator SAW
-------------------------------------------
The supervisor of a job writes the record, and the coordinator reads that record
twice each second. A job that is shorter than that period thus gives `starting`
and then `completed`, WITH NO `running` LINE. The field `previous` of that line
says `starting`, so the sequence that you read is the true sequence.
The stream gives no line for a state that the coordinator did not see. A line
for such a state would be a statement that qex cannot support.
Read the stream again after a stop
----------------------------------
qex events --json --since <stream_id>:348 # the events after 348
qex events --json --since start # everything that it holds
qex events --json --since now # the new events only
The default is `start`.
KEEP TWO VALUES: the `stream_id` of the first line, and the largest `seq` that
you read. Give both to `--since` when your program starts again, as
`<stream_id>:<seq>`. You then lose nothing while the same coordinator operates.
THE NUMBERS BELONG TO ONE COORDINATOR. The coordinator stops when no job
operates, and the next command starts a new one. That coordinator starts its
numbers at 1 again, and it makes one event for each record that it reads. Your
number 348 thus names a DIFFERENT event there.
With the stream name, qex compares the two and gives you a `gap` line that says
that the coordinator changed, then continues with the events that the new
coordinator holds. Its job records are the same records.
A NEW COORDINATOR THUS GIVES YOU SOME LINES A SECOND TIME. It makes one event
for each record that it reads, so a job that stopped while you were away arrives
again as `completed`, with a new number. This is the ordinary case: the
coordinator retires when no job operates, which is when your program is away.
ACT ON `id` AND `state`, AND NOT ON THE ARRIVAL OF A LINE. Keep the states that
you acted on, by job id, and do the work of a line one time.
WITH A NUMBER ALONE, qex cannot make that comparison, and you can lose events
with no message. Give the name. `qex events` writes a warning when you give a
number with no name.
What happens when you do not read fast enough
---------------------------------------------
The coordinator keeps the last 512 events. It NEVER waits for a reader, and it
never grows its memory for one. If you do not read the stream fast enough, the
coordinator drops the oldest events and you receive a `gap` line that COUNTS
them. Do the work of an event in a different thread or process, and keep the
reader reading.
qex reports a gap. It does not hide one, because a reader that loses `failed`
and hears nothing waits for a result that will never arrive.
The field `missed` counts the events. It is `null` when qex cannot count them,
which occurs when your number comes from a different stream: the two streams
have no common measure, so a number there would say something that qex cannot
support. The `reason` field says what happened.
The end of the stream
---------------------
The coordinator stops when no job operates and no command arrives for one hour.
A reader does NOT hold it open. Before it stops, it writes a `bye` line, and the
command then exits with the code 0.
If the stream ends with NO `bye` line, something stopped the coordinator. The
command writes a message to stderr and exits with the code 1. The records of the
jobs are on the disk and they are correct; run the command again to read the
stream of the next coordinator.
Options
-------
--json one JSON object for each line. Use this option.
--since VALUE `start`, `now`, `<stream_id>:<seq>`, or a bare `seq`
--count N stop after N events
--timeout TIME stop after this time. The exit code is then 124.
An earlier coordinator
----------------------
A coordinator that operates can be older than this command. Such a coordinator
does not know this request, and `qex events` REFUSES to run: it names the
coordinator, and it gives the command that stops it. It never gives you an empty
stream, because an empty stream and a stream with no events look the same.
";
pub const OUTPUT: &str = "\
qex output and files
====================
JSON
----
Each command that reads data accepts `--json`. The output is one JSON document.
qex list --json
qex status <id> --json
qex wait <id> --json
For the schema of these documents:
qex schema status
qex schema job
qex schema event
Job files on the disk
---------------------
qex writes one directory for each job:
~/.local/state/qex/jobs/<uuid>/
spec.json the command, the environment and the claims (mode 0600)
status.json the state, the exit code, the times and the true use
stdout.log the standard output of the job
stderr.log the standard error of the job
A job that writes more than `[logs] max_bytes` also has `stdout.log.tail` or
`stderr.log.tail` while it operates. That file holds the last part of the
output, and qex writes it into the log file and deletes it when the job stops.
The directory has mode 0700 because `spec.json` can contain secrets.
`status.json` is the primary record. The supervisor of the job writes it in one
operation, so a reader sees the old contents or the new contents. `qex wait`
reads this file directly if the coordinator does not operate.
Logs
----
`qex logs` and `qex status` accept the same options to select lines.
qex logs <id> both streams, the last 500 lines
qex logs <id> --all every line
qex logs <id> --stdout one stream
qex logs <id> --tail 100 the last 100 lines
qex logs <id> --head 20 the first 20 lines; a fault at the start
qex logs <id> --lines 400:430 the lines from 400 to 430
qex logs <id> --number write the line number before each line
qex logs <id> --grep ERROR the lines that match
qex logs <id> --grep E -C 3 with 3 lines before and after each match
qex logs <id> --grep x --fixed read the value as plain text
qex logs <id> --max-matches 20 show 20 matches, and count the others
qex logs <id> --follow the output while the job operates
qex logs <id> --follow --tail 50 the last 50 lines, then the new lines
qex logs <id> --hook the output of the stop hook, and the verdict
of qex on it
qex logs <id> --follow --grep ERROR the matches as they arrive
Every path has a limit. A search reports the number of lines that match, so a
pattern that matches 3000 lines tells you that the pattern is too wide.
The output of a job also has a limit
------------------------------------
A job can write more than `[logs] max_bytes` (the default is 32MB for each
stream). qex then keeps the first part of the output and the last part, and it
writes one line between them:
[qex] ---- 361MB and 4201177 line(s) of the output are not in this file ----
Those lines are NOT on the disk. `--all` does not give them back, because
nothing holds them. `qex status` and `qex logs` say how much went, and
`qex status --json` gives the numbers in the field `logs_dropped`.
qex removes nothing until the output passes the limit. A job that writes less
than the limit, less the room that qex keeps for the notes (2KB), keeps every
byte in one piece, and a second attempt of a job that failed keeps the output of
the first attempt. Above that point, a job that passes the limit by one byte
gets the same file as a job that passes it by a gigabyte: qex writes the file
while the job runs, and at that moment nobody knows how much output follows.
The log file becomes shorter at the moment that the output passes the limit.
`qex logs --follow` says so, and it continues at the new end of the file.
Make `[logs] max_bytes` larger for a job that must keep everything, or write
the output of the job to a file of your own.
The output of a job is a pipe
-----------------------------
The supervisor reads the output through a pipe and writes the file itself. The
standard output and the standard error of a job are thus a pipe, and not a
regular file. Almost every program sees no difference. Three things change:
lseek gives ESPIPE, and stat gives a FIFO in place of a regular file. A
program that asks for its position in its own output meets an error.
Two children of one job that write more than 4096 bytes in one operation
can mix in the middle of a line. A regular file kept each write
together.
isatty gives false, as it did before.
If a program needs a regular file, give it one:
qex submit -- sh -c 'my-program > out.txt'
A pipe closes when the last process that holds it stops, so a job that leaves a
process behind (`setsid`, `nohup ... &`, a daemon that a test starts) keeps its
output open after the job ends. qex waits 30 seconds for the output to close and
then writes the result: a record that arrives is worth more than a wait with no
end. The record then says `incomplete` in the field `logs_dropped`, and `error`
says that a log file can be missing its last part. The wait does not fail the
job. To get the result at once, give that process an output of its own:
qex submit -- sh -c 'setsid my-daemon > daemon.log 2>&1 &'
Use `--follow --grep` in place of a pipe to `grep`. A pipe holds the lines in a
buffer and shows nothing until the buffer fills, because `grep` needs the option
`--line-buffered`. qex writes each line as it reads it.
Watch the queue
---------------
qex events --json one line for each change of state, as it happens.
Use this command in a program. See `qex help events`.
qex top the jobs, the claim of each one, and its true use now
qex top --once one page, for a script
qex top -i 5 a refresh every 5 seconds
The CPU column gives the cores in use. Compare it with the CPU CLAIM column to
find a claim that is much larger than the need.
This command never starts a coordinator, and it gives the jobs when no
coordinator operates.
Delete the records
------------------
qex clean <id> one job
qex clean completed each job that succeeded
qex clean done each job that stopped
qex clean --state failed each job in one state
qex clean --cwd the jobs of this directory
qex clean --under the jobs of this directory and below
qex clean --under /path the jobs of that directory and below
qex clean --auto a short form of `--state done
--older-than 1h`, on this directory and
below. A job of the last hour stays,
because it is frequently the job that you
read now.
qex gc every record of every directory that
stopped more than one day ago. It also
deletes a job directory that holds no
record. Use `--dry-run` first, and
`[gc] keep` to change the time.
qex du how much disk space qex holds, and the
job records that hold the most
`qex list` takes `--cwd` and `--under` as well, so you can see what a deletion
would remove.
A job that a job in the queue still needs is NOT finished for a deletion,
whatever its own state says. The job in the queue reads that record to decide
whether to run, and to explain why it did not. `qex clean` and `qex gc` keep
such a record and say so, and it goes when the other job stops.
qex clean --older-than 7d each job older than 7 days
qex clean --all every job
`qex clean` deletes the directory of the job. It does not stop a job that
operates.
qex keeps the id of a deleted job for one day, so `qex status` can tell you that
a job existed and that its work happened. An agent thus does not repeat work
after a deletion. Change that time with `[history] keep` in the config file.
`qex clean --all` deletes the record of EVERY job of this user, including the
jobs of a different agent that shares this machine. Use `qex clean <id>` when
another agent uses qex at the same time.
";
pub const PIPELINE: &str = "\
qex pipelines
=============
A pipeline file describes several jobs, and one command submits them all. The
key in the file is `[[jobs]]`, and each entry becomes a qex job with its own id,
its own record and its own log file. This text says `job` for that reason.
qex pipeline ci.toml
The command writes the group id to stdout, and the id of each stage to stderr,
so `GROUP=$(qex pipeline ci.toml)` operates.
name = \"ci\"
[[jobs]]
name = \"build\"
command = [\"make\"]
[[jobs]]
name = \"unit\"
command = [\"make\", \"test\"]
needs = [\"build\"]
[[jobs]]
name = \"lint\"
command = [\"make\", \"lint\"]
needs = [\"build\"]
[[jobs]]
name = \"ship\"
command = [\"./deploy.sh\"]
needs = [\"unit\", \"lint\"]
[[jobs]]
name = \"cleanup\"
command = [\"./clean.sh\"]
after = [\"ship\"]
Each job in the file takes every field of a job file: `cwd`, `env`, `timeout`,
`max_queue_time`, `tags`, `priority`, `env_capture`, `nice`, `locks`, `retries` and
`[resources]`. A stage thus claims a GPU in the same way as a job file:
[[jobs]]
name = \"train\"
command = [\"uv\", \"run\", \"train.py\"]
needs = [\"build\"]
[jobs.resources]
cpu = 4
mem = \"16GB\"
gpu = 1
vram = \"20GB\"
A stage that waits for an earlier stage also uses its `max_queue_time`, because
that clock counts every wait. Give a value that covers the whole pipeline, or
give no value on such a stage.
Why a pipeline file, and not several submissions
------------------------------------------------
A name is easy to write, and a name is not unique in time. If you run the same
four jobs twice with `qex submit --needs build`, that name gives two jobs.
The names in a pipeline file belong to that file and to that one submission.
qex changes each one into the id that it made a moment before, and no name
leaves the file. A second run of the same file makes new jobs with new ids, and
the two runs never meet.
One command for the whole pipeline
----------------------------------
Every job of one submission shares a group id, and that id names every stage:
GROUP=$(qex pipeline ci.toml)
qex wait $GROUP # wait for every stage
qex status $GROUP # the state of every stage
qex kill $GROUP # stop every stage
qex clean $GROUP # delete every record
qex list --group $GROUP
The name of the pipeline works in the same way as its id, with one limit: a
pipeline takes its name from its file, so a second run of that file has the same
name. qex refuses a name that gives two runs, and it shows the group id of each.
Use the group id in a script.
`qex status --json` gives an array for a pipeline and one object for one job,
so a script that reads one job does not change. A pipeline of one stage still
gives an array, because the shape comes from what you named.
`qex logs` reads one job, so it refuses a pipeline and names the stages.
Use `--id-file` to keep every id in a file:
qex pipeline ci.toml --id-file ids.env
. ids.env # gives $group, $build, $unit, ...
qex status \"$ship\" --wait
A name that ends in `.json` gives a JSON object instead, for a parser.
qex reads the whole file before it submits anything. A circle of jobs, a name
that no job has, and a job with no command each give an error, and no job
starts.
For one command and many inputs, use `--each-line`. Run `qex help each-line`.
";
pub const EACH_LINE: &str = "\
qex fan-out: one job for each line
==================================
One command, many inputs. `--each-line` reads a file and submits one job for
each line. Put `{}` in the command, and each job gets the text of one line
there.
qex submit --each-line inputs.txt -- ./process {}
The jobs share one group id. The group id goes to stdout, and the name and id of
each job go to stderr, so this operates:
GROUP=$(qex submit --each-line inputs.txt -- ./process {})
qex list --group $GROUP
Read the lines from another program with the name `-`:
ls *.parquet | qex submit --each-line - -- ./convert {}
Where `{}` goes
---------------
`{}` goes in any argument, in the program name, or inside an argument:
qex submit --each-line urls.txt -- curl -o {}.html https://{}/
Every `{}` takes the line. A command with NO `{}` gives an error, because each
job would then be the same command and the lines would have no effect.
Write `{{}}` for a literal `{}`. Nothing else in the command changes.
A line is data, and never a command
-----------------------------------
qex starts no shell. Each line becomes exactly ONE argument, whatever it holds:
a space, a quotation mark, a semicolon, a dollar sign or a newline. A file of
names from a directory listing or from a database is therefore safe.
a b\"; rm -rf ~; echo $HOME
That line gives one argument with those characters in it. No shell reads it.
To use a shell feature, name the shell, and give the line as an argument and
never inside the text of the script:
qex submit --each-line names.txt -- bash -c 'echo \"$1\" | tr a-z A-Z' _ {}
A line that starts with a dash
------------------------------
A line becomes an argument, so a line such as `-v` or `--out=/etc/passwd`
becomes an OPTION of your program. qex cannot know which arguments your program
reads as options, so it does not change the line.
Put `--` in the command before `{}`. Almost every program then reads the line as
data and not as an option:
qex submit --each-line names.txt -- ./process -- {}
This is the same rule as `xargs`. Use it for input that you did not write
yourself.
Which lines give a job
----------------------
a line one job
an empty line no job
a line that starts # no job, it is a comment
the space at each end qex removes it
a CRLF ending the same job as an LF ending
no final newline the last line still gives a job
qex says on stderr how many lines it passed over, so a line that you expected
never goes away in silence.
A file that is not UTF-8 gives an error with the line number, and qex submits
nothing. The command of a job is text, so qex cannot run such a line.
All or nothing, and the one case that is not
-------------------------------------------
qex reads the whole input and tests the command first. Every fault that qex can
find gives an error and NO job at all.
One case remains: qex submits the jobs one at a time, so a coordinator that
stops in the middle leaves the earlier jobs in the queue. qex then writes the
group id and the id of every job that it submitted, and how to stop them. Read
the group id from that message, because stdout holds the group id of a fan-out
that succeeded in full.
The limits
----------
`--each-line` submits 1000 jobs at most. Each job holds a directory, so a file
with 100000 lines would fill the disk. Raise the limit when you need it:
qex submit --each-line big.txt --max-jobs 5000 -- ./process {}
qex reads 64 MiB at most, from a file and from a pipe, because it holds the
whole input in memory. `--max-jobs` does not raise that limit.
The options that a fan-out refuses
----------------------------------
--dedupe-key, --dedupe-window A key holds ONE job. Every job of the
fan-out would carry the same key, so qex
would start the first line only.
--json It writes the id of one job. Use
`--id-file NAME.json` for the group and
every job.
--job The place for the line belongs on the
command line, where a reader sees it.
`qex run` does not accept `--each-line` at all. It waits for ONE job.
The name of each job
--------------------
Each job gets a name for `qex list`: the program name, the position in the
file, and as much of the line as fits.
process-01-data-a.csv
process-02-data-b.csv
Give `--name` to change the first part and the name of the group.
A name holds the letters, the numbers, `.`, `_` and `-` only. A line can hold a
terminal control sequence, and a name goes to your terminal in `qex list`.
The other options
-----------------
`--cpu`, `--mem`, `--timeout`, `--max-queue-time`, `--lock`, `--tag`,
`--priority`, `--env`, `--nice`, `--needs`, `--after` and `--retries` apply to
every job of the fan-out.
qex calculates the claim one time, from the command of the FIRST line, and
gives it to every job. The lines of a fan-out are the same kind of work.
`--max-queue-time` is the time that ONE job waits, and not the time of the
group. The jobs that still wait at the end of that time become `expired`.
N jobs make N of everything
---------------------------
`qex events` writes at least 3 lines for each job, and the coordinator holds
the last 512 events only. Start `qex events` BEFORE you submit a large fan-out,
and filter on `.job.group`. A reader that falls behind receives a `gap` line
with the number of events that it lost.
`[hooks] on_stop` runs one time for EACH job that stops, and that includes
`expired` and `skipped`. A fan-out of 1000 lines runs the hook 1000 times. The
hook environment names the job and not the group, so give the fan-out a `--tag`
and read `QEX_TAGS`.
A fan-out learns as one task
----------------------------
qex records what each job used, and gives that measurement to the next job of
the same command. A fan-out does not fit that rule: `./process a.csv` and
`./process b.csv` are two commands, and each one runs one time.
qex therefore measures every job of a fan-out against the TEMPLATE
`./process {}`. One fan-out makes one record, and the second run of the same
fan-out gets its claim from the first run.
`qex status` says `(from the earlier jobs of this fan-out)` for such a claim.
Use `--lock` when the jobs must not operate together:
qex submit --each-line inputs.txt --lock db -- ./load {}
Keep the ids in a file
----------------------
qex submit --each-line inputs.txt --id-file ids.env -- ./process {}
. ids.env # gives $group and one name for each job
A name that ends in `.json` gives a JSON object instead, for a parser.
";
pub const EXIT_CODES: &str = "\
qex exit codes
==============
`qex wait`
----------
0 the job succeeded (exit code 0)
1 the job failed (a different exit code, or a signal)
123 the job never started. It waited more time than its
`--max-queue-time` value, and its state is `expired`.
124 your wait timed out. The job still operates.
125 something stopped the job: kill, cancel, timeout or out-of-memory
126 the job did not run, because a job that it needed did not succeed
127 there is no job with that id
The code 124 has the same meaning as the code of the `timeout` command.
The code 123 is not 125. A job with the code 125 ran and wrote output. A job with
the code 123 never got the machine, so it has no output. Read the `error` field
of `qex status` for the wait that stopped it.
A timeout on `qex wait` stops your wait only. It does not stop the job. Use
`qex kill` to stop the job.
To get the exit code of the job itself, add `--passthrough`:
qex wait $ID --passthrough
`qex wait` then exits with the exit code of the job. Use this option to send the
result of the job to a script.
`qex run`
---------
the exit code of the job the job ran (0, 7, 1, whatever it gave)
123 the job never started; it reached its `--max-queue-time`
124 your wait stopped, and the job continues. See the dedupe key below.
125 something stopped the job: kill, cancel, Ctrl-C, timeout, out-of-memory
126 the job did not run, because a job that it needed did not succeed
127 there is no job with that id
`qex run` writes the output of the job, so it gives the exit code of the job
when the job RAN. `qex run -- sh -c 'exit 7'` gives 7.
A job of `qex run` is a job like any other, so `qex kill` and `qex cancel` from
a DIFFERENT command can stop it. That job gave no exit code of its own, and
`qex run` then gives 125 and not 1. The two are thus separate: 125 says that
something stopped your work before it could finish. `qex run` also writes a line
to stderr that names the cause, and that line says when this command did not
stop the job.
The code 1 has two causes. Your work ran and it gave the exit code 1, or qex
could not finish its own work: the coordinator stopped while `qex run` waited,
for example. qex writes the second cause on stderr, and the job can then still
operate.
For each state in which the job gave NO exit code of its own, `qex run` gives
the same code as `qex wait`. Two commands must not answer one question two ways.
For a job that RAN, `qex run` gives the exit code of the job, and `qex wait`
gives 0 or 1 unless you add `--passthrough`.
`qex run` gives 124 in ONE case: a dedupe key gave it the job of a different
caller, and a signal then arrived. This command did not start that job, so
Ctrl-C stops this wait and the job continues. The code 124 says the same thing
there as on `qex wait`: YOUR WAIT ended, and the work did not. Run
`qex status $ID --wait` to wait again, or `qex kill $ID` to stop the job.
`qex run` gives 124 for no other reason. It waits with no limit of its own, and
a job that reaches the time limit of `--timeout` gives 125, because something
stopped that job.
Other commands
--------------
0 the command succeeded
1 the command failed
2 the command line is not correct
127 there is no job with that id
";