Module runpod

Expand description

Runpod as a training target: a pod created for a run over the REST API (v2), reached over SSH with per-run keys, guarded by a watchdog running on the pod, and deleted once the run’s results are retrieved.

A Runpod target is not an executor of its own: once its pod is ready, the run goes through the SshExecutor and the native runtime, like an SSH target.

Structs§

Assessment: What one look at the disk calls for.
Attempt: One create call.
Capture: The pod’s log kept in a run directory (POD_LOG, mode 0600) and its cursor (POD_LOG_CURSOR). Past POD_LOG_CAP, one line says the log was capped and nothing more is kept. The cursor is written when it moved, at most once a second while lines keep coming, then once more at the end (dropping the capture writes it too): a cursor behind the log repeats a few lines on the next resume, never loses one.
CostCap: When the cost cap of a pod applies, in Unix seconds.
CreatePod: The body of POST /pods: exactly the v2 fields overbrainer sets (the API rejects any unknown field with a 422).
CudaVersion: A CUDA version offered by a GPU type’s machines.
DataCenter: A data center of the catalog (GET /catalog/datacenters).
DataCenterList: GET /catalog/datacenters.
DataCenterStock: A data center with the stock of chosen GPU types.
DiskProbe: What disk_probe_script found.
DiskWatch: The disk policy of one followed run: it warns from WARN_PERCENT on, and at ACT_PERCENT, or when the next checkpoint would not fit, grows the network volume within max_volume_gb or stops the job with a snapshot.
GpuFilter: What select_gpus keeps. The default keeps every Secure Cloud GPU type.
GpuMaxCount: Most GPUs of one type a pod can have, per cloud: a ceiling, not stock.
GpuOffer: A GPU type in stock and what a pod of it costs.
GpuPrice: List prices of one GPU, in USD per hour.
GpuRequest: The GPU part of CreatePod.
GpuType: A GPU type of the catalog (GET /catalog/gpus). Only the fields overbrainer uses are read; every one tolerates null.
GpuTypeList: GET /catalog/gpus.
InvalidPodId: A string rejected by PodId::new: empty, or holding anything but ASCII letters and digits.
LogQuery: What to ask of the log stream.
Mounts: Volumes of CreatePod.
NetworkMount: A network volume and where it is mounted.
NetworkVolume: A network volume of the account (GET /network-volumes).
NetworkVolumeList: GET /network-volumes.
NewSecret: The body of POST /account/secrets. Only the JSON serializer ever sees the value: Debug shows the name only.
Pagination: Paging of GET /pods and GET /templates.
Pod: A pod, as GET /pods/{id}, GET /pods and POST /pods return it. Only the fields overbrainer uses are read.
PodCtx: What the pod flows share.
PodEnv: The part of a pod’s environment overbrainer reads back.
PodGpu: The GPUs of a pod.
PodId: A pod ID as Runpod assigns it: letters and digits only, so it is safe in a URL path and a file.
PodKeys: The keys of a run: the client key overbrainer logs in with, and the pod’s host key. The private host key only lives in memory, as the base64 of its OpenSSH file, until it is sent in the create call.
PodLogLine: One line of a pod’s log, as the stream gives it and POD_LOG keeps it.
PodPage: One page of GET /pods.
PodPlan: What to create for a run.
PodRecord: runs/<run-id>/pod.json. It never holds a private key, the API key or the pod’s environment: only the public host key and what Runpod reported.
PodRow: A row of overbrainer pod ls.
PodSettings: What a pod’s bootstrap and watchdog need to know, sent in its env.
PodSsh: SSH access to a pod.
Provisioned: A pod ready for the run: reachable over SSH with its pinned host key, with a watchdog that proved it can delete it.
Removal: What pod rm did: the pods it deleted, even when it failed.
Removed: A pod pod rm deleted.
RunpodClient: A client of the Runpod REST API (v2) with the account API key.
RunpodTarget: A runpod target of overbrainer.toml, defaults applied.
Secret: A secret of the account (/account/secrets). Runpod never returns its value.
SecretList: The answer of GET /account/secrets.
SshDirect: The public endpoint of a pod’s SSH server.
SshEndpoint: The SSH endpoint of a pod as last seen.
Stock: The stock of one catalog entry (a GPU type in a data center, or a data center for a GPU type).
Template: A template of the account (GET /templates). Its env, which may hold secrets, is never read.
TemplatePage: One page of GET /templates.
Timing: How long the provisioning steps wait. Timing::standard in production; tests use milliseconds.
Usage: How much of the run’s disk is used.
VolumeDisk: The network volume of a pod.

Enums§

ApiError: Errors of the Runpod API. No variant ever holds the API key or a pod’s host key. A create call’s error never holds any text from Runpod’s answer either, however that text was shaped: its message is chosen only from the HTTP status (see RunpodClient::status_error), because no amount of pattern matching against ways a server might chunk, escape or encode an echoed value can be shown complete; not showing any of it is the only claim this client can make and keep.
AttemptResult: How one create call ended.
Availability: A catalog stock band, as Runpod writes it.
Critical: Why the disk must be acted on.
DeleteReason: Why overbrainer deletes a pod.
DeletedBy: Who removed a pod.
Ending: What became of a pod once its job ended.
LogError: Why reading a pod’s log stopped.
LogSource: Where a line of a pod’s log comes from.
PodError: Errors of a Runpod run’s pod: provisioning, keys, readiness, deletion.
PodState: Where a run’s pod stands.
PodStatus: A step of a pod’s life.
RemoteStatus: What Runpod says a pod is doing (desiredStatus in v1, status in v2).
ResolveError: Why an auto choice of a Runpod target found nothing: each message says what was asked.
RowKind: What overbrainer knows of a pod of pod ls.
Watched: How watch_on_pod ended.

Constants§

ACT_PERCENT: Use, in percent, at which the disk is grown or the job stopped.
BOOTSTRAP_LOG: The bootstrap’s log on the pod, in the run directory on the pod.
CLIENT_KEY: The run’s client private key, in SSH_DIR.
COST_CAP_FILE: Where the client writes, on the pod, when the watchdog deletes the pod for the cost cap (Unix seconds), in the run directory.
DEADLINE_MARGIN: How long after the watchdog’s deadline the client deletes the pod itself, when it is still there: the watchdog should have done it first.
DISK_PROBE_EVERY: How often, at most, the disk probe (du) runs.
DRAIN_WAIT: How long the last read before a delete waits for the lines not read yet.
GROW_WINDOW: How long the API may keep showing the old size after a grow before the job is stopped.
HOST_KEY_ENV: The create call’s variable holding the base64 of the pod’s private host key: a reference to the run’s host key secret, which Runpod resolves at boot.
HOST_KEY_SECRET_PREFIX: Prefix of the name of every run’s host key secret.
JOB_ENV: Where the bootstrap writes the jobs’ environment on the pod.
KNOWN_HOSTS: The pod’s pinned host key, in SSH_DIR.
LEASE_FILE: The lease a client following a job keeps on its pod, in the run directory: the watchdog skips its max_hours deadline while the file was touched less than LEASE_TTL ago.
LEASE_TTL: How long a renewed lease holds the deadline off; the watchdog’s OVERBRAINER_LEASE_TTL default.
MAX_COST_REACHED: Why a run fails when its pod was deleted on max_cost_usd by its watchdog.
MAX_HOURS_REACHED: Why a run fails when its pod was deleted on max_hours, by the client’s own guard or by the pod’s watchdog.
MIN_CAP_TIME: Least time max_cost_usd should buy for a run to train before its cost snapshot: SNAPSHOT_LEAD twice.
MIN_CUDA_VERSION: Lowest CUDA version of the host driver, for DEFAULT_RUNPOD_IMAGE’s CUDA 13.
POD_FILE: The pod record of a run, in its local run directory.
POD_LOG: The pod’s logs, in a run directory: JSON lines {ts, source, line}.
POD_LOG_CAP: Largest POD_LOG kept, in bytes: past it, one line says so and the capture stops.
POD_LOG_CURSOR: The last event ID of the stream kept in POD_LOG, in a run directory.
POD_RECORD_VERSION: Format version of POD_FILE.
RETRIEVED_MARKER: Where the watchdog looks for the client’s “retrieved” marker, in a run directory on the pod.
SECRETS_FORBIDDEN_MESSAGE: A secret write was refused with a 403: the API key lacks the permission.
SNAPSHOT_AT_FILE: Where the client writes, on the pod, when the job is stopped with a snapshot for the cost cap (Unix seconds), in the run directory: read by the watchdog.
SNAPSHOT_LEAD: How long before the pod is deleted at the latest its job is asked for a snapshot: the watchdog’s OVERBRAINER_SNAPSHOT_LEAD default, for the cost cap as for the deadline.
SNAPSHOT_SHARE: Share of max_cost_usd at which the job is stopped with a snapshot.
SSH_CONFIG: The run’s ssh config, in SSH_DIR.
SSH_DIR: Directory of a run’s SSH files, in its local run directory.
TAIL_MAX: Most lines Runpod replays (tail); more is refused.
USER_AGENT: User-Agent of every request. Runpod sits behind Cloudflare, which answers a generic client’s user agent with a 403 (“error code: 1010”).
VOLUME_MOUNT: Where a network volume is mounted.
VOLUME_SIZE_FILE: Where the client writes the network volume’s size in GB, in the run directory, for the watchdog’s own disk rule.
VOLUME_USABLE_PERCENT: Share of a network volume’s size, in percent, that can really be written: Runpod’s quota refuses writes a little before the size in GB.
VOLUME_WORKDIR: Directory of the run directories on a network volume, so they outlive the pod.
WARN_PERCENT: Use, in percent, of the first warning.
WARN_STEP: Percent between two warnings, after the first one.
WATCHDOG_ACT_PERCENT: Use, in percent, at which the watchdog asks for a snapshot on its own: later than ACT_PERCENT, so a grow overbrainer is making comes first.
WATCHDOG_LOG: The watchdog’s log on the pod, in the run directory on the pod.
WORKDIR: Directory of the run directories on a pod without a network volume. Nothing else in the image uses it, and /workspace itself is never a mount point.

Functions§

alias: The SSH host alias of a run’s pod, also its HostKeyAlias.
arm_cost_cap: Hands the cost cap of pod to its watchdog before the job of run starts: when to stop the job with a snapshot and when to delete the pod, in files the watchdog reads every minute, with a warning when the cap buys too little training (short_cap_warning). Nothing to do without a cap, or for a kept pod.
assess: What usage calls for: a warning at WARN_PERCENT and at each further WARN_STEP above warned (the last warning’s percent), and an action at ACT_PERCENT or when the free space is less than 1.5 times checkpoint_bytes, the size of the newest checkpoint when known.
base64: bytes in standard base64, with padding and no line break.
bootstrap_functions: The bootstrap’s shell functions, without the call that runs them.
by_price: Cheapest Secure Cloud price first (no price last), ties by more VRAM, then by ID.
capture: Keeps the log of the pod pod_id in the directory of the run run_id while the pod exists: follows its stream from the kept cursor (or the longest replay), until the pod is gone, the cap is reached or the API refuses. Best effort: failures are logged, never returned.
chain: error and its sources, joined with : .
cheapest_in_stock: The cheapest GPU type in stock a pod of target would get, from the catalog gpus, priced for target.gpu_count GPUs, as overbrainer pod gpus lists prices: its auto GPU types as resolve_with_floor picks them (with floor_gb), else its listed ones. None when none is in stock with a price.
connect_followed: Connects to the pod of the run run with client through the ssh config the process following it wrote (the endpoint pod.json holds, for the built-in client), for train stop while that process holds the project: no Runpod API call, and nothing written locally, so the run directory stays that process’s alone. None when the run has no pod left or no ssh config or endpoint yet.
data_center_stock: Every data center with the stock of each chosen GPU type, ordered by ID.
data_center_table: The rows of overbrainer pod datacenters, header first, sorted by ID; nothing when data_centers is empty. The stock column counts the GPU types data_centers reports in stock there, not a chosen set.
disk_probe_script: The script printing the size of the newest checkpoint of the run in run_dir, and with a network volume mounted at volume_dir, what the volume holds, both in KiB, each after its @name line. It always succeeds.
drain: The last read of the log of the pod pod_id into the directory of the run run_id, before the pod is deleted: what the stream gives from the kept cursor within DRAIN_WAIT. Best effort.
drop_host_key: Deletes the host key secret of the run run_id, found by name; nothing when there is none.
drop_host_key_or_warn: drop_host_key, a failure only warned about with how to finish it.
end_pod: Ends the pod of the run run once its job ended, after copying the watchdog’s and the bootstrap’s logs into runs/<run-id>/.pod/. Only when the results were retrieved (Outcome::retrieved: every file the pod has was downloaded and verified) is the “retrieved” marker written on the pod, first (so its watchdog deletes it even if this client’s delete fails), then the pod is deleted unless kept and its deletion confirmed, other pods of the run are swept (any whose deletion cannot be confirmed is recorded in pod.json), and, when none is, the run’s private client key is removed and its host key secret deleted. Otherwise the pod is left to its watchdog’s retrieve grace (a kept pod stays kept), with a warning telling how to retrieve the results again or remove the pod.
export_room_warning: The warning when the container disk of container_disk_gb GB, without a network volume, may be too small for an export needing need_bytes (see export_disk_bytes) beside the run: more than 90% of the disk. None when it fits.
follow: Follows the started run record with runs::watch, until its job ends, with the client’s own guard: at the watchdog’s deadline plus DEADLINE_MARGIN, a pod still there is deleted and the run saved Failed (never for a kept pod). That is watch_on_pod, the pod’s logs kept meanwhile (with_pod_logs), then settle_watch.
follow_logs: Follows the log stream of the pod id for query, forever: sink gets each batch of lines as snapshot says. A dropped connection is opened again from the last event ID, after 0.5 s doubling up to 15 s (reset once an event arrives), or Runpod’s Retry-After. Returns only why it stopped: a refusal that would come back on every reconnect (400, 401, 403, 404, 422; a 404 once the pod is deleted), or a failed sink.
forget_client_key: Removes the private client key of a run whose pod is gone, best effort.
forget_keys: Removes the private client key of the run run_id and deletes its host key secret, best effort, once its pod.json shows no pod of it that may still exist (no pod, or one confirmed deleted, and no stray): a stray pod restarted later boots with the same secret. Callers save pod.json first, strays included. Keys kept here go once the strays are deleted: by overbrainer pod rm, or by the sweep of the secrets of ended runs at the next start.
gpu_table: The rows of overbrainer pod gpus, header first, for gpus already filtered and sorted (see select_gpus); nothing when gpus is empty. data_center, when given, is shown as the stock column instead of the overall band (see GpuType::stock_in).
grown_size: The size, in GB, to grow a network volume of current_gb to: half again its size, and at least 50 GB more, but never past max_gb. None when that is no growth at all.
host_key_placeholder: What the pod’s env holds instead of the host key: Runpod replaces it with the secret’s value when the pod boots.
host_key_run: The run of the host key secret name, or None when name is not one.
host_key_secret: The name of the host key secret of the run run_id. A run ID is ASCII letters, digits, _ and -, at most 64 characters, so the name meets Runpod’s rules (a letter first, at most 191 characters).
hours: max_hours as a duration, zero when it cannot be one (the configuration rejects such values).
job_started: Records that the job of the run was started on the pod.
keep_file: Replaces the kept file path of run_dir with bytes, whole or not at all: written to a new temporary file (mode 0600, a stale one removed first), then renamed over it.
kept_cursor: The event ID the log kept in run_dir ends at, if any.
kept_lines: The lines of the log kept in run_dir, oldest first; a line that cannot be read is skipped. Empty when none is kept.
limit_reached: Why the run of pod, found gone at now, failed when a limit of its watchdog had passed: MAX_COST_REACHED or MAX_HOURS_REACHED.
listed_rows: The rows of the pods that one list of the account shows, as pod_rows makes them, with no look at any single pod and nothing recorded: what the orphan warning of train is built from. A recorded pod the list does not show has no row: it is not billing, or pod ls finds it.
one_line: text on one line: a newline and a carriage return are written \n and \r, other control characters but tab, and the bidirectional overrides and isolates, \u{..}.
open_kept: Opens the kept file path (such as POD_LOG or BOOTSTRAP_LOG) of run_dir for reading; None when it, or .pod, does not exist.
orphan_warnings: One warning per orphan pod among rows, saying how to remove it. Never deletes anything.
parse_disk_probe: Reads the answer of disk_probe_script.
parse_kept_line: A line of the kept log, cleaned like a line of the stream (see PodLogLine::cleaned); None when it is not one.
past_deadline: Whether the watchdog’s own deadline for pod has passed at now; never for a kept pod, or one without a deadline. A pod found gone then was deleted by its watchdog on max_hours.
pod_command: The pod’s cmd: bash -c with the bootstrap, a write_watchdog function holding the watchdog in a quoted here-document, and the call to bootstrap_main. write_watchdog writes to a sibling .tmp file and only mv -fs it into place once the write itself succeeded, so a write that fails partway (a full disk) can never leave fail a truncated watchdog to exec into.
pod_endpoint: Writes the run’s ssh config (see write_config) and returns what reaches the pod at endpoint, with either client: the config for OpenSSH, the endpoint, client key and pinned host key for the built-in client.
pod_env: The pod’s env. It holds no secret: the host key is only a reference to the run’s host key secret (see host_key_placeholder), HF_TOKEN never goes there (the job gets it over SSH), nor does the account API key (the pod has its own pod-scoped key).
pod_rows: Every pod of the account whose run marker is set or whose name starts with overbrainer-, plus every pod recorded in runs/ (the run’s pod and its stray_pods) that the list does not show. Those are looked up together (see look_up_all) and declared gone only after three 404s in a row and a list without them (never by a delete): the run’s pod is then recorded deleted in pod.json, and a stray is dropped from stray_pods. A pod.json that cannot be read is skipped with a warning.
printable: text, from the Runpod API, without what could move the cursor or change the terminal’s state: ANSI escape sequences (CSI, string controls such as OSC or DCS through their terminator, and two-character ones) are dropped, a control character that is whitespace (newline, tab, CR…) becomes a space, and every other one (C0, DEL, C1) is dropped.
provision: Creates the run’s pod, trying plan.target.gpu_types in order (its auto choices resolved first, see resolve_target), and waits until it is ready. With auto GPU types, up to three cheaper types reported out of stock are tried before those in stock, and the catalog is read again after each type that cannot be placed, so the types not tried yet are sorted again. Before the first create call, the pod’s host key is stored as the run’s Runpod secret (see store_host_key), which every pod of the run refers to; the caller deletes it once no pod of the run is left. record (pod.json) is saved before every create call and after every answer. A pod that dies or stays unreachable is deleted and the next GPU type tried. When provisioning fails after a create call got no clear answer, every pod of the run still listed is deleted (see sweep).
read_kept: The whole kept file path of run_dir (see open_kept); None when it does not exist.
read_kept_from: The bytes of the kept log of run_dir from offset, or from its start when it is now shorter, with where they start; None when none is kept.
reconnect: Connects again to the pod of the run run with client, for train attach and train cancel: looks it up (its public port may have changed), rewrites the run’s ssh config and connects. None when the pod no longer exists (confirmed by repeated looks, never by a delete), which pod.json then records.
remove: Deletes the run’s current pod and waits until the API no longer knows it, after a last read of its logs into the run directory (see super::drain), then records it deleted by by, or by DeletedBy::Watchdog when it was already gone. Does nothing when there is no pod. The delete is sent even when pod.json cannot be saved first.
remove_run_pods: Deletes the pods of the run run_id: the one in its pod.json (even kept), its stray_pods, and any other carrying its marker. Every delete is sent and confirmed; a stray is dropped from stray_pods once gone, and a pod found by its marker whose delete fails is added to it. Every pod is tried; the first failure is returned at the end.
resolve: target with its auto choices replaced by lists, from the catalog’s gpus, listed for the target’s gpu_count; listed choices are kept as they are.
resolve_target: target with its auto choices resolved from the Runpod catalog (see resolve_with_floor, with floor_gb), logged at info level; a target without any is returned as it is, without an API call. Both choices are resolved from the same GPU listing, scoped to target.gpu_count: catalog/datacenters is never read for this.
resolve_with_floor: resolve, auto GPU types keeping only those with at least floor_gb of VRAM (the estimate of what the run needs, see crate::train::sizing) when the target sets no min_vram_gb: an explicit min_vram_gb wins.
select_gpus: The GPU types filter keeps, cheapest first (see by_price).
settle_watch: Acts on what watch_on_pod found for the run run_id: past the deadline or the cost cap, deletes the pod and fails the run; after a failed watch, fails the run when its pod is gone. Callers must not drop it midway (the CLI shields it from Ctrl-C), since it may be deleting the pod.
short_cap_warning: A warning when cap USD buys less than MIN_CAP_TIME of a pod billed rate USD an hour: its job is stopped with a snapshot SNAPSHOT_LEAD before the cap, so it would hardly train. None otherwise, and without a positive rate and cap.
snapshot: Reads what the log stream of the pod id gives for query and stops once it has been silent for 750 ms after its first event, once the server closes it, or after max_wait in all (Runpod sends nothing, headers included, until it has a line: a pod with nothing new ends there). sink gets each batch of lines, redacted, with the event ID they end at. Returns the last event ID seen, or query’s own.
ssh_command: The command reaching a kept pod of the run or export run_id of runs: ssh -F runs/<id>/ssh/config overbrainer-<id>.
ssh_config: The text of a run’s ssh config.
start_pod: Creates the pod of the new run run (from runs::create) and waits until it is ready: keys in runs/<id>/ssh/, then pod.json, then provisioning. When that fails, the run is saved Failed with the reason (interrupted before the job started after Ctrl-C), whatever pod was created is deleted, and once every pod of the run is confirmed deleted, the run’s private client key is removed and its host key secret deleted. vram_floor_gb is the least VRAM auto GPU types need when the target sets no min_vram_gb (see PodPlan::vram_floor_gb).
stocked_data_centers: The data centers with at least one chosen GPU type in stock, ordered by the cheapest such GPU type they have (see by_price), ties by ID.
store_host_key: Stores value, the base64 of the pod’s private host key, as the host key secret of the run run_id: created, or, when the name is taken (a create retried after an answer that got lost), found by name and given the value. From then on the client redacts the value too.
sweep: Deletes every pod carrying the marker of run_id but keep: a pod left over by an ambiguous create that showed up late. Returns the pods whose deletion could not be confirmed, for pod.json, or None when the pods could not be listed, so nothing is known about leftovers.
sweep_host_keys: Sweeps the run host key secrets (overbrainer_host_key_<run>, or overbrainer_host_key_<export-id> for an export’s pod) that no pod needs any more, given rows, the pods of one list of the account (see listed_rows): a secret whose run has a listed pod is kept, and so is one of a run of this project still in progress (it may be creating its pod) or whose record cannot be read. The secret of an ended run of this project is deleted; one of a run that is not in this project’s runs/ only gets a warning saying how to remove it, as its pods would, since another checkout may be starting that run. Returns the warnings.
table: The rows as the aligned table pod ls prints, header first; nothing when there is no row.
template_table: The rows of overbrainer pod templates, header first, sorted by name; nothing when templates is empty.
volume_table: The rows of overbrainer pod volumes, header first, sorted by name; nothing when volumes is empty.
watch_leased: watch_on_pod for a client that stays with the job: while the job keeps making progress (a metric in the last 30 minutes), it renews the pod’s lease (LEASE_FILE) every few minutes, so max_hours deletes nothing, neither the watchdog nor the client’s own guard. Once the job stalls or the lease cannot be renewed, the lease runs out after LEASE_TTL and the deadline applies again. It warns once, as information, when the job ends after the deadline. Once the pod has spent SNAPSHOT_SHARE of its cost cap, it asks the job for a snapshot, as the watchdog does. A kept pod has no deadline and needs no lease. The disk policy runs beside it, kept pod or not.
watch_on_pod: Watches the started run record until its job ends or the client’s deadline (the watchdog’s plus DEADLINE_MARGIN; none for a kept pod) passes, and warns once when the job’s pace cannot end it before the watchdog’s deadline. The disk policy runs beside it. It never deletes anything, so dropping it (on Ctrl-C) loses nothing: what it found is acted on by settle_watch.
watchdog_script: The watchdog as the pod runs it: the group_signal helper of the job scripts, then watchdog.sh.
with_pod_logs: Runs work while the log of the pod of pod is kept in its run directory; the capture never ends work early. Only work when no pod is recorded or it is deleted.
write_config: Writes the ssh config of a run into dir (created if needed), with endpoint as the pod’s address, and the known_hosts it points to. Rewritten before every connection, since a pod reset changes its public port. Returns the config’s path.
write_known_hosts: Writes known_hosts into dir: one line pinning host_public for alias. Returns its path.

Module runpod

Module runpod Copy item path

Structs§

Enums§

Constants§

Functions§

Module runpod