aion_server/control/

pid_file.rs

//! The server's pid file: `<AION_HOME>/run/aion-server.pid`.
//!
//! The file is the stop/status verbs' address for the running server. It is
//! written at BIRTH — before the store is opened, before anything can take
//! minutes — and it carries the incarnation's whole life: the state it is in
//! ([`IncarnationState`]), the boot stage it is working through, the
//! addresses once they are bound, and the drain window once it is known.
//!
//! RULING — the record appears at birth, not at bind. A big store takes
//! minutes of WAL recovery, and a record written only at bind left every
//! control verb blind for that whole window: `status` said "no server has
//! claimed this home", `stop` said "nothing to stop" (exit 0), and the
//! launcher's port probe read the home as empty and spawned ANOTHER server,
//! which blocked silently on the store writer lock. Four servers stacked
//! invisibly on 2026-08-26. The birth claim is what makes that impossible:
//! the second boot's [`claim_at_birth`](super::claim::claim_at_birth) sees a
//! live booting sibling and REFUSES.
//!
//! Every mutation of the file — the birth claim's rename-into-place, each
//! stage write, the bind-time fill, the drain flip, the stop verb's
//! compare-and-delete, the guard's own exit-time compare-and-delete — runs
//! under an exclusive OS file lock ([`std::fs::File::lock`]: `flock` on
//! Unix, `LockFileEx` on Windows) on a sibling lock file
//! (`aion-server.pid.lock`). The lock is what makes compare-and-delete
//! atomic: without it, a successor claiming the home between a remover's
//! read and its unlink would have ITS record deleted, leaving a live server
//! no verb can address. The lock file itself is never renamed or removed —
//! locking the pid file directly would be unsound, because the claim
//! replaces that path's inode and a lock on the old inode excludes nobody.
//!
//! RULING — a lock failure at claim time REFUSES the boot. The lock is the
//! instrument that keeps one home's records from destroying each other; a
//! home where it cannot be taken (permissions on the lock file, a filesystem
//! without advisory locking) is a home where a claim can silently delete a
//! live server's address, and the honest answer is a refusal naming the
//! remedy, not a boot that runs without the guarantee. The exit-time guard
//! is deliberately more lenient (it leaves the file for stale
//! reconciliation) because at that point refusing helps nobody — the
//! process is exiting either way.
//!
//! RULING — every future field added to [`PidRecord`] is `#[serde(default)]`
//! on read. The record is a cross-process, cross-version contract: a newer
//! CLI must be able to read the record an older running server wrote —
//! upgrade time is exactly when the stop verb matters most — so a missing
//! field reads as its honest default, never as a parse refusal.
//!
//! RULING — records are compared by INCARNATION IDENTITY (pid + start
//! instant), never by whole-record equality. The record now MUTATES during
//! the incarnation's life (every stage write bumps `stage_seq`), so a
//! whole-record compare would make the stop verb's reconciliation and the
//! guard's exit-time delete miss their own record the moment a stage landed
//! between the read and the compare — leaving live-looking debris behind
//! every boot that took long enough to report progress.
//!
//! The record is an incarnation identity, never a bare pid: pid, start
//! instant, and the serving binary's content hash (plus the bound addresses
//! and build identity, so readers talk to the server that IS running rather
//! than the one today's config would start). A stale file found at claim
//! time is reconciled by incarnation check and reported — never silently
//! overwritten, never trusted. See [`crate::control::incarnation`] for how a
//! record is verified against the live process table.

use std::net::SocketAddr;
use std::path::{Path, PathBuf};

use serde::{Deserialize, Serialize};

use crate::error::ServerError;

/// File name of the pid file inside the Aion home's `run/` directory.
const PID_FILE_NAME: &str = "aion-server.pid";

/// File name of the mutation lock beside the pid file. Held exclusively for
/// the duration of every pid-file mutation; never renamed, never removed
/// (unlinking a lock file reintroduces the race the lock exists to close).
const PID_LOCK_FILE_NAME: &str = "aion-server.pid.lock";

/// Take the exclusive pid-file mutation lock for `pid_path`'s home.
///
/// Blocks until the lock is granted. Holders are short but not instantaneous:
/// the longest section is [`claim_at_birth`](super::claim::claim_at_birth)'s,
/// which spans the stale-record reconciliation — including an incarnation
/// probe that refreshes the process-table instrument, tens to hundreds of
/// milliseconds under load — so waiting is bounded by that, not by a single
/// syscall.
///
/// The lock is released when the returned [`std::fs::File`] drops: closing
/// the descriptor releases both `flock` (Unix) and `LockFileEx` (Windows)
/// locks, and neither release path can fail or panic.
///
/// # Errors
///
/// Returns [`ServerError::PidFile`] when the lock file cannot be created or
/// the lock cannot be taken.
pub(super) fn lock_pid_mutation(pid_path: &Path) -> Result<std::fs::File, ServerError> {
    let lock_path = pid_path.with_file_name(PID_LOCK_FILE_NAME);
    let file = std::fs::OpenOptions::new()
        .create(true)
        .truncate(false)
        .write(true)
        .open(&lock_path)
        .map_err(|io_error| {
            pid_file_error(format!(
                "could not open the pid mutation lock `{}`: {io_error}",
                lock_path.display()
            ))
        })?;
    file.lock().map_err(|io_error| {
        pid_file_error(format!(
            "could not take the pid mutation lock `{}`: {io_error}",
            lock_path.display()
        ))
    })?;
    Ok(file)
}

/// Where a recorded incarnation is in its own life.
///
/// The three states are observable facts about a process, not policy: it is
/// working through its boot ([`Booting`](Self::Booting)), it has both doors
/// open ([`Serving`](Self::Serving)), or it has seen a termination signal and
/// is draining ([`Draining`](Self::Draining)). Every one of them is a state a
/// control verb must be able to address — that is the whole point of writing
/// the record at birth.
#[derive(Clone, Copy, Debug, Default, Deserialize, Eq, PartialEq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum IncarnationState {
    /// The process has claimed the home and is working through its boot: the
    /// store is opening, WAL recovery is replaying, the engine is recovering
    /// resident workflows. No listener is bound yet, so the addresses are
    /// `None` and the doors are shut.
    Booting,
    /// Both transport listeners are bound and the server is serving. This is
    /// the `#[serde(default)]` value on read (module-header ruling): a record
    /// written by a build that predates this field only ever existed AFTER
    /// the bind, so `Serving` is what it always meant.
    #[default]
    Serving,
    /// A termination signal has been observed and the drain is running (or
    /// about to). The record stays in place — a draining server is exactly
    /// the server `aion server status` must still describe — and a successor
    /// boot SUCCEEDS it rather than being refused.
    Draining,
}

impl IncarnationState {
    /// The state's operator-facing name, uppercase, as the verbs render it.
    #[must_use]
    pub const fn label(self) -> &'static str {
        match self {
            Self::Booting => "BOOTING",
            Self::Serving => "SERVING",
            Self::Draining => "DRAINING",
        }
    }
}

/// The pid file's content: one JSON object describing the server incarnation
/// that claimed the home.
#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)]
pub struct PidRecord {
    /// Operating-system process id of the serving process.
    pub pid: u32,
    /// The process start instant in whole seconds since the Unix epoch, read
    /// through the process-table instrument ([`sysinfo`]) at boot. Verifiers
    /// compare this against the SAME instrument's answer for the live pid, so
    /// the check is exact equality, not clock arithmetic: a reused pid wears
    /// a different start instant.
    pub started_at_unix_secs: u64,
    /// SHA-256 of the serving binary's bytes, hashed from the executable path
    /// at boot. This records WHICH build claimed the home; it is not a
    /// liveness discriminator (an upgrade that swaps the binary by rename
    /// changes the file on disk while the process keeps its old image).
    pub binary_sha256: String,
    /// Crate version of the serving binary (`CARGO_PKG_VERSION` at build).
    pub version: String,
    /// Source commit of the serving binary, from
    /// [`crate::build_identity::BuildIdentity`].
    pub commit: String,
    /// Where this incarnation is in its own life. Defaulted on read
    /// (module-header ruling) to [`IncarnationState::Serving`].
    #[serde(default)]
    pub state: IncarnationState,
    /// HTTP address the server actually bound — recorded so `status` probes
    /// the running server's own listener even when the config file has been
    /// edited since boot. `None` while the incarnation is still
    /// [`Booting`](IncarnationState::Booting): nothing is bound yet, and an
    /// address recorded before the bind would be a promise, not a fact.
    #[serde(default)]
    pub http_address: Option<SocketAddr>,
    /// gRPC address the server actually bound, recorded for the same reason
    /// and `None` for the same window.
    #[serde(default)]
    pub grpc_address: Option<SocketAddr>,
    /// HTTP address this incarnation's already-loaded configuration says it
    /// WILL bind — a promise by name, recorded at birth precisely because it
    /// is one. The bound pair above stays the fact about open doors; this
    /// pair exists so a SECOND boot arriving mid-recovery can decide
    /// collision against the holder's configuration instead of presuming it:
    /// two explicit configs on one home with different doors are two servers,
    /// and the second must boot unclaimed rather than refuse. `None` only in
    /// a record written by a build that predates the field, where the honest
    /// reading remains "unknowable, presume collision".
    #[serde(default)]
    pub intended_http_address: Option<SocketAddr>,
    /// gRPC counterpart of `intended_http_address`, same birth, same reason.
    #[serde(default)]
    pub intended_grpc_address: Option<SocketAddr>,
    /// The boot stage this incarnation is working through, as a machine
    /// token: `config`, `store-open`, `writer-lock-wait`, `wal-recovery`,
    /// `engine-recovery`, `binding`. `None` once the boot is done.
    #[serde(default)]
    pub stage: Option<String>,
    /// One human sentence about the current stage — "materializing shard 17
    /// of 64", "waiting 40s on the store writer lock at <path>". The token
    /// above is what a script keys on; this is what an operator reads.
    #[serde(default)]
    pub stage_detail: Option<String>,
    /// Incremented on EVERY stage write. Readers watch it to distinguish a
    /// boot that is PROGRESSING from one that is STUCK — two facts a bare
    /// stage token cannot tell apart, and the difference between "wait" and
    /// "something is wrong".
    #[serde(default)]
    pub stage_seq: u64,
    /// When the last stage write landed, in whole seconds since the Unix
    /// epoch. Wall-clock, because the reader is a different process: a
    /// monotonic instant does not cross the process boundary.
    #[serde(default)]
    pub stage_updated_at_unix_secs: u64,
    /// The drain window this incarnation is actually running with, in whole
    /// seconds — recorded for the same reason as the addresses: the running
    /// server's window may have come from its own command line
    /// (`--drain-timeout`) or from a config file edited since boot, and a
    /// stop verb that waited out a re-derived value would misreport a healthy
    /// long drain as "still draining" (or force it on a second invocation).
    /// Defaulted on read (module-header ruling): a record written by a build
    /// that predates this field — or by an incarnation still BOOTING, which
    /// has not resolved its runtime config yet — reads as 0, which patience
    /// resolution treats as "no recorded window" and falls through to config.
    #[serde(default)]
    pub drain_timeout_seconds: u64,
}

impl PidRecord {
    /// This record's incarnation identity: the pid and the start instant,
    /// which together name exactly one process on exactly one boot of the
    /// machine.
    ///
    /// This — never whole-record equality — is what every compare-and-act
    /// path uses (module-header ruling). The rest of the record mutates
    /// while the incarnation lives.
    #[must_use]
    pub const fn identity(&self) -> (u32, u64) {
        (self.pid, self.started_at_unix_secs)
    }

    /// Whether `self` and `other` name the same incarnation.
    #[must_use]
    pub fn is_same_incarnation(&self, other: &Self) -> bool {
        self.identity() == other.identity()
    }

    /// How long ago this incarnation started, in whole seconds, against the
    /// reader's own wall clock. Saturating: a record from a machine whose
    /// clock has since moved backwards reports 0 rather than an absurdity.
    #[must_use]
    pub fn running_for_secs(&self) -> u64 {
        now_unix_secs().saturating_sub(self.started_at_unix_secs)
    }

    /// How long ago the last stage write landed, in whole seconds, or `None`
    /// when no stage has ever been written (a record from a build predating
    /// stages, or an incarnation that never reported one).
    #[must_use]
    pub fn stage_age_secs(&self) -> Option<u64> {
        if self.stage_updated_at_unix_secs == 0 {
            return None;
        }
        Some(now_unix_secs().saturating_sub(self.stage_updated_at_unix_secs))
    }

    /// The stage rendered for an operator: `wal-recovery — materializing
    /// shard 17 of 64`, or just the token when no detail was written, or
    /// `None` when no stage has been reported at all.
    #[must_use]
    pub fn stage_line(&self) -> Option<String> {
        let stage = self.stage.as_ref()?;
        match &self.stage_detail {
            Some(detail) => Some(format!("{stage} — {detail}")),
            None => Some(stage.clone()),
        }
    }
}

/// Whole seconds since the Unix epoch, or 0 when the system clock is set
/// before the epoch (which no reader can do arithmetic on anyway).
///
/// Public because the record's stage timestamps are wall-clock by necessity
/// (they cross a process boundary), so every reader that renders an age has
/// to read the same clock the writer did — through this one function, rather
/// than each growing its own epoch arithmetic.
#[must_use]
pub fn now_unix_secs() -> u64 {
    std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .map_or(0, |elapsed| elapsed.as_secs())
}

/// What the birth claim found already sitting at the pid-file path,
/// reconciled before the new record was written. Reported to the operator
/// through the boot log; returned so tests can assert the reconciliation
/// happened rather than trusting the log.
///
/// One face is not a reconciliation at all: [`Collision`](Self::Collision)
/// is what the birth claim converts into a refusal
/// ([`HomeAlreadyClaimed`](super::claim::HomeAlreadyClaimed)). It lives in
/// this enum because it is one of the answers "what is already here?" has,
/// and keeping it out would have meant a second classification pass.
#[derive(Clone, Debug, Eq, PartialEq)]
pub enum StaleReconciliation {
    /// No file was present: the ordinary boot.
    NonePresent,
    /// A file was present, its pid is alive, the incarnation MATCHES, it is
    /// BOOTING or SERVING, and its doors collide with the ones this boot
    /// intends to open. Nothing is written and the boot is REFUSED: at birth
    /// no bind has happened, so there is no bind-success argument to overrule
    /// a live record with, and starting anyway is exactly the invisible
    /// stacking the birth claim exists to prevent.
    Collision(PidRecord),
    /// A file was present but its process is gone — the previous server died
    /// without removing its record (crash, `SIGKILL`). The record is replaced.
    DeadIncarnation(PidRecord),
    /// A file was present and its pid is alive, but the live process's start
    /// instant does not match the record — the pid was reused by something
    /// else after the recorded server died. The record is replaced; the
    /// stranger keeps running untouched.
    ReusedPid(PidRecord),
    /// A file was present, its pid is alive, the incarnation MATCHES — but
    /// it is recorded on DIFFERENT addresses from the ones this boot INTENDS
    /// to bind, so the two servers do not collide: that server may be serving
    /// its addresses right now, and this one is about to serve others. The
    /// claimant boots UNCLAIMED: the first claimant keeps the record and the
    /// control verbs, the new incarnation serves without them, and the
    /// warning names the consequence. Neither server's record is ever
    /// destroyed by the other's boot or exit.
    LiveIncarnationElsewhere(PidRecord),
    /// A file was present, its pid is alive, the incarnation MATCHES, and it
    /// is DRAINING: the recorded server has seen its termination signal and
    /// is on its way out. This boot SUCCEEDS it — the record is replaced with
    /// the successor's, and the handover is logged naming both incarnations.
    /// The drainer's own guard leaves the successor's record alone at exit
    /// (its compare is by incarnation identity, which no longer matches).
    SucceededDrainer(PidRecord),
    /// A file was present but could not be parsed as a [`PidRecord`] (a
    /// hand-written pid from the pre-verb ritual, or corruption). Replaced.
    Unreadable {
        /// Why the existing content did not parse.
        reason: String,
    },
}

/// Absolute path of the pid file under `home`.
#[must_use]
pub fn pid_file_path(home: &Path) -> PathBuf {
    home.join("run").join(PID_FILE_NAME)
}

/// Read and parse the pid file under `home`.
///
/// Returns `Ok(None)` when the file does not exist — the ordinary "no server
/// has claimed this home" state, distinct from every error.
///
/// # Errors
///
/// Returns [`ServerError::PidFile`] when the file exists but cannot be read,
/// or exists and cannot be parsed as a [`PidRecord`].
pub fn read(home: &Path) -> Result<Option<PidRecord>, ServerError> {
    read_path(&pid_file_path(home))
}

/// [`read`] against an explicit path.
///
/// # Errors
///
/// Returns [`ServerError::PidFile`] when the file exists but cannot be read
/// or parsed.
pub(super) fn read_path(path: &Path) -> Result<Option<PidRecord>, ServerError> {
    let content = match std::fs::read_to_string(path) {
        Ok(content) => content,
        Err(io_error) if io_error.kind() == std::io::ErrorKind::NotFound => return Ok(None),
        Err(io_error) => {
            return Err(pid_file_error(format!(
                "could not read pid file `{}`: {io_error}",
                path.display()
            )));
        }
    };
    let record = serde_json::from_str::<PidRecord>(&content).map_err(|parse_error| {
        pid_file_error(format!(
            "pid file `{}` exists but does not parse as a pid record: {parse_error}. \
             If it was written by hand (the pre-verb restart ritual wrote a bare pid), \
             remove it and restart the server so the server writes its own record",
            path.display()
        ))
    })?;
    Ok(Some(record))
}

/// Write `record` to `path` atomically: staged into a sibling temp file, then
/// renamed into place. The caller MUST already hold the mutation lock.
///
/// # Errors
///
/// Returns [`ServerError::PidFile`] when the record cannot be serialized, the
/// staging file cannot be written, or the rename fails.
pub(super) fn write_record_atomically(path: &Path, record: &PidRecord) -> Result<(), ServerError> {
    let content = serde_json::to_string(record).map_err(|serialize_error| {
        pid_file_error(format!(
            "could not serialize the pid record: {serialize_error}"
        ))
    })?;
    let temp_path = path.with_extension("pid.tmp");
    std::fs::write(&temp_path, format!("{content}\n")).map_err(|io_error| {
        pid_file_error(format!(
            "could not write pid file staging `{}`: {io_error}",
            temp_path.display()
        ))
    })?;
    std::fs::rename(&temp_path, path).map_err(|io_error| {
        pid_file_error(format!(
            "could not move pid file into place at `{}`: {io_error}",
            path.display()
        ))
    })
}

/// Remove the pid file under `home` if — and only if — it still holds the
/// SAME INCARNATION as `record`. Used by the stop verb to reconcile the file
/// a non-clean exit (forced, killed) left behind, after the process is proven
/// gone.
///
/// The compare is on incarnation identity, never whole-record equality
/// (module-header ruling): the caller's copy was read before the signal, and
/// a server that reported a boot stage in between has a record that differs
/// in `stage_seq` while naming the very same process. A whole-record compare
/// would leave that server's record behind forever.
///
/// Returns whether the file was removed.
///
/// # Errors
///
/// Returns [`ServerError::PidFile`] when the file cannot be read or removed.
pub fn remove_if_matches(home: &Path, record: &PidRecord) -> Result<bool, ServerError> {
    let path = pid_file_path(home);
    if !path.exists() {
        // No pid file means nothing to remove and — since the lock file lives
        // beside it — possibly no run directory to open a lock in either.
        return Ok(false);
    }
    // Read, compare, and unlink under the mutation lock: without it a
    // successor claiming the home between the read and the unlink would have
    // ITS record deleted, leaving a live server no verb can address.
    let mutation_lock = lock_pid_mutation(&path)?;
    let removed = match read_path(&path)? {
        Some(current) if current.is_same_incarnation(record) => {
            std::fs::remove_file(&path).map_err(|io_error| {
                pid_file_error(format!(
                    "could not remove pid file `{}`: {io_error}",
                    path.display()
                ))
            })?;
            true
        }
        Some(_) | None => false,
    };
    drop(mutation_lock);
    Ok(removed)
}

/// Build the [`ServerError`] for a pid-file failure.
pub(super) fn pid_file_error(message: impl Into<String>) -> ServerError {
    ServerError::PidFile {
        message: message.into(),
    }
}

#[cfg(test)]
#[path = "pid_file_tests.rs"]
mod tests;