Skip to main content

Module execution_recovery

Module execution_recovery 

Source
Expand description

Launch-time recovery for killed detached docker sessions (issue #176).

$ --isolated docker --detached --on-kill-resume 3 --recovery-command B -- A runs A; when the main process is killed (exit 137, or OOMKilled with no exit status of its own), the detached completion watcher hands the inspected facts to this module, which restarts the same container so B continues on the same filesystem, with the same resource limits (they live in the container’s HostConfig), under the same execution UUID and appending to the same log file.

How B replaces A inside the same container: a container launched with a recovery command runs a tiny selector as its command. The selector runs A normally, but runs B when the marker file exists. Before docker start, the marker (holding the attempt number) is copied into the stopped container with docker cp. Without a recovery command the original command is simply started again.

Like detached_finalize, the entry point runs as a short-lived child of the watcher on a host whose CLI invocation is long gone, so it never panics on bad input and never prints anything.

Structs§

RecoveryDelayHooks
Where the --on-kill-resume-delay wait gets its randomness and sleeps; tests swap both out (issue #181).
RecoveryFacts
The docker facts the watcher hands over.
RecoveryOutcome
Outcome of a recovery attempt; recovered is true only when the container was started again.
RecoverySeparator
What the [Recovery k/N] separator reports.

Constants§

INTERNAL_RECOVER_FLAG
Hidden argument the detached watcher re-invokes this binary with.
KILLED_EXIT_CODE
Exit code of a process killed with SIGKILL (the OOM killer’s signal).
RECOVERY_ATTEMPT_ENV
Environment variable carrying the attempt number into the recovery command.
RECOVERY_MARKER_PATH
File whose presence makes the selector run the recovery command.

Functions§

build_recovery_selector_args
Wrap the container command so a later docker start can switch to the recovery command.
build_recovery_snippet
Shell condition run by the completion watcher: true when the container was killed and the recovery entry point resumed it. Mirrors is_killed_exit: OOMKilled only counts without a non-negative exit code (issue #178).
format_recovery_separator
The [Recovery k/N] separator written into the session log.
is_killed_exit
Whether the inspected facts describe a killed main process.
recover_killed_execution
Resume a killed execution in its own container.
recover_killed_execution_with_delay
recover_killed_execution after the random --on-kill-resume-delay wait (issue #181), with injectable randomness and sleep.
recovery_selector
POSIX selector run as the container command when a recovery command is set: sh -c SELECTOR start-command <shell> <flag> <recovery> <main argv...>.
run_internal_recover
Entry point for --internal-recover-detached-docker <uuid> <exit> <oom> <started> <finished> <error> <cgroup>. Returns 0 only when the container was resumed; any other outcome lets the watcher continue with its normal cleanup, footer and finalization.
start_system_watcher
The real watcher: a detached shell following the resumed container.

Type Aliases§

WatcherStarter
Starts the completion watcher that follows a resumed container.