Expand description
cgroup v2 memory counters of a detached docker container (issue #182).
Docker’s State.OOMKilled is container-wide and sticky (moby/moby#43564):
it says that some process in the container was OOM-killed at some point,
but not how many, not whether the container hit its own --memory limit or
the whole host ran out, and not how close the execution came to its limit.
The kernel keeps all three per cgroup
(https://docs.kernel.org/admin-guide/cgroup-v2.html#memory-interface-files):
memory.events `oom` the cgroup hit its own limit
memory.events `oom_kill` processes killed here by *any* OOM killer, so
`oom_kill > oom` points to a host-wide (or parent
cgroup) OOM
memory.peak / memory.max peak usage (Linux 5.19+) and limit (`max` when
unlimited)The cgroup is removed when the container stops, so the values cannot be read
afterwards. The detached completion watcher therefore samples them while the
container runs and hands the last sample to the post-mortem, the finalizer
and the recovery step. Best effort by design: no cgroup v2, a remote docker
daemon or a hidden /proc leave the counters unknown, and a kill in the last
sampling interval can be missed when the cgroup is gone before the final
read.
Modules§
- cgroup_
shell_ vars - Shell variables used by the sampler snippets.
Structs§
- Cgroup
Memory - The counters of one sample.
limit_bytesisNonewithout a limit.
Enums§
- OomScope
- Where the OOM killer that killed processes in the container came from.
Constants§
- CGROUP_
SAMPLE_ INTERVAL_ SECONDS - Seconds between two samples of a running container’s cgroup.
Functions§
- build_
cgroup_ functions_ snippet - Shell functions shared by the sampler and its final read:
- build_
cgroup_ memory_ log_ snippet - Shell fragment appending the
Memory:line of the post-mortem when a sample exists.quoted_log_pathmust already be shell-quoted. - build_
cgroup_ sampler_ start_ snippet - Shell fragment starting the background sampler. It writes each sample to a
temp file with
mv, so the watcher never reads a half-written line, and ends by itself once the cgroup is gone. - build_
cgroup_ sampler_ stop_ snippet - Shell fragment stopping the sampler once the container has exited and
leaving the last sample (
<max> <peak> <oom> <oom_kill>, or empty) in$__start_command_cgroup. The cgroup can outlive the main process for a moment, so it is read one last time when it still exists. - describe_
cgroup_ oom_ scope Nonewhen nothing was OOM-killed.- deserialize_
cgroup_ memory deserialize_withhelper forExecutionRecord::cgroup_memory.- format_
cgroup_ memory - Human-readable counters for
--status, e.g.peak 255.9 MiB of 256.0 MiB limit, oom 0, oom_kill 3 (...). - format_
cgroup_ memory_ log_ line - The
Memory:line for a sample, exactly as the watcher’s shell writes it. Used where Rust writes the post-mortem itself (the recovery step). - normalize_
cgroup_ memory - Normalize counters read back from an execution record. Lenient, like the JavaScript reader: a malformed value never makes the record unreadable.
- parse_
cgroup_ memory_ sample - Parse a sample written by the watcher
(
<memory.max> <memory.peak> <oom> <oom_kill>);Nonewithout a sample.