Module system_memory 

System-memory probing and memory-derived download buffer defaults.

The download buffer configuration defaults (see config::groups::reconstruction) are derived from the memory actually usable by this process: the minimum of the host’s physical RAM and the effective cgroup memory limit. Inside a container, the cgroup limit is what matters; System::total_memory (and psutil in Python) report the host total, which is how a 1 GiB pod would otherwise size a multi-GB buffer.

The cgroup limit comes from sysinfo’s Process::cgroup_limits (sysinfo >= 0.39), which resolves the cgroup path listed in /proc/self/cgroup rather than reading the controller root, and takes the tightest bound across ancestor cgroups. That covers cgroup v2 and v1, nested paths (a Slurm step, a systemd slice), and namespaced container roots.

Structs

DownloadBufferDefaults

The three download buffer knobs, derived together so they describe one allocation.

UsableMemory

Memory available to this process, as best we can determine.

Constants

STATIC_DEFAULTS

Static fallback defaults, used when memory cannot be determined or derivation is disabled via HF_XET_DISABLE_MEMORY_DERIVED_DOWNLOAD_BUFFERS=1. These are the historical constant defaults.

STATIC_HP_DEFAULTS

Static fallback for high-performance mode; the historical HF_XET_HP constants. These values also serve as the derivation ceilings below.

Statics

USABLE_MEMORY

The system memory reading, probed once and cached for the process lifetime.

Functions

default_download_buffer_sizes

The memory-derived defaults for the standard configuration.

high_performance_download_buffer_sizes

The memory-derived defaults for high-performance mode (HF_XET_HP).

log_derived_defaults

Log the probed memory and the derived download buffer defaults.