gunnar_server/

clock.rs

//! **The appliance's wall clock**: glue between PID 1 and edda `estate-clock`.
//!
//! Every rule — the sources, the quorum, the sanity window, the signed-peer
//! protocol, step-at-boot and slew-while-running, the first fed syncs and the
//! running triggers — lives in edda's `estate-clock` crate (where Gunnar's own
//! `gunnar-clock` moved, lane L10 2026-10-21), which the front's gunnar-front
//! links too. This file decides only *whether* this process owns the clock,
//! *where* its feed is, and *what it says* about the answer.
//!
//! **Both wires during the move**: a box asks in `ESTATE-TIME/1` and asks again
//! in `GUNNAR-TIME/1` when an old front answers in Gunnar's dialect
//! (`estate_clock::peer::query`); `GUNNAR_CLOCK*` (the kit's
//! `korp.env.GUNNAR_CLOCK=`) is read as an alias of `ESTATE_CLOCK*`
//! (`estate_clock::config::var`).
//!
//! # What went wrong with the first cut (gunnar `5792dda`), measured
//!
//! The appliance booted two hours fast; `5792dda` asked three internet HTTPS
//! hosts for a `Date` at boot and was re-imaged on 2026-09-14 — and the box
//! still refused credentials `7199529 ms away`. Through the control plane, with
//! a credential deliberately signed 7 199 500 ms fast so the gate let it
//! through, `gunnar control mirror add … https://github.com/…` answered
//! `github.com could not be resolved: failed to lookup address information: Try
//! again`. UpCloud's firewall catch-all `drop` drops the replies to outbound
//! UDP (from the front: `dig +notcp` times out, `dig +tcp` answers), musl's
//! resolver asks over UDP only, so every source was unreachable, the verdict was
//! "no quorum", and the clock was left alone. The only record was a log line on a
//! VGA console. See the `estate-clock` crate note for the full measurement.
//!
//! # What it does now
//!
//! * **The source is configuration** (`korp.env.GUNNAR_CLOCK=` on the kernel
//!   command line). On gunnar.rs that is `peer://10.13.8.99:50123` — the front,
//!   by IP literal over the utility network, answering a nonce with a signed
//!   timestamp (`GUNNAR_CLOCK_KEY` names its public key). No internet, no DNS.
//!   Unset means **not configured**, and nothing is asked: an appliance in an
//!   airgap has no business dialling Stripe for the time.
//! * **At boot**, after the network and before any listener: one sync, steps.
//! * **While running**, only when a credential is refused for skew (rate-limited
//!   to one sync a minute) and on a 6-hour safety timer — `estate_clock::trigger`.
//!   Small offsets slew, large ones step — `estate_clock::adjust`.
//! * **Every attempt is recorded** (`estate_clock::report`) and served on the
//!   control plane: `Admin.Health` carries the boot and the last report, and
//!   `Bootstrap.Status` — the one call that needs no credential — carries the
//!   last one and this server's own clock. When the clock is wrong a credential
//!   is exactly what gets refused, so the credential-free call is the one that
//!   makes a wrong clock diagnosable from the front.
//!
//! # Why it can never stop the boot or the server
//!
//! Every function here returns `()` and swallows everything. A forge that
//! refuses to come up because a clock fetch failed is strictly worse than a
//! forge with a wrong clock: the wrong clock refuses credentials, the dead
//! forge refuses everything — including the call an operator uses to find out
//! why. The periodic sync runs on the blocking pool, never on a request path.

use estate_clock::config::{Config, CLOCK_ENV};
use estate_clock::offer::FedRun;
use estate_clock::report::{Outcome, Report, Trigger};

pub use estate_clock::offer::{last_outcome, outcome_is_right};

/// **One sync** (AIRGAP, owner rule 2026-10-25). With `clock-fetch` it is estate-clock's own,
/// HTTPS Date sources included. Without it (the default build) the same rule over peer and
/// SNTP sources only: an `https://` source is refused by [`NO_HTTPS`] and never dialled.
#[cfg(feature = "clock-fetch")]
fn sync(config: &Result<Config, String>, phase: Trigger) -> Report {
    estate_clock::fetch::sync(config, phase)
}

/// The refusal an `https://` clock source gets in a build without `clock-fetch`.
pub const NO_HTTPS: &str =
    "an https:// clock source needs gunnar-server's `clock-fetch` feature (off by default: this build asks peer and SNTP sources only)";

#[cfg(not(feature = "clock-fetch"))]
fn sync(config: &Result<Config, String>, phase: Trigger) -> Report {
    let report = match config {
        Ok(Config::Sources { sources, peer_key }) => {
            let polled = estate_clock::poll::poll_with(sources, peer_key.as_deref(), &|_| {
                Err(NO_HTTPS.to_owned())
            });
            let action = estate_clock::adjust::choose(phase, &polled.verdict.decision);
            let applied = estate_clock::adjust::apply(action).map_err(|err| err.to_string());
            Report::from_poll(phase, &polled, action, applied)
        }
        Err(why) => Report::without_sources(phase, Outcome::Misconfigured, why.clone()),
        Ok(Config::NotConfigured) => Report::without_sources(
            phase,
            Outcome::NotConfigured,
            format!("no clock source is configured ({CLOCK_ENV})"),
        ),
        Ok(Config::Off) => Report::without_sources(
            phase,
            Outcome::Off,
            format!("{CLOCK_ENV}=off: this machine was told not to set its own clock"),
        ),
    };
    estate_clock::report::record(report.clone());
    report
}

/// Whether this process may set the machine's clock at all.
///
/// Gated exactly as `crate::disktest` and `crate::dashboard` are — PID 1 **and**
/// `korp.entry=` on the kernel command line. A container entrypoint is PID 1 of
/// its namespace too, and a privileged container shares the host's clock:
/// stepping it there would move somebody's real machine. Naming sources in
/// [`CLOCK_ENV`] (or its Gunnar alias `GUNNAR_CLOCK`) explicitly turns it on
/// regardless, which is what the installtest rig and a developer reproducing a
/// refusal need; the syscall still needs `CAP_SYS_TIME` and a refusal is still
/// only a report.
fn owns_the_clock() -> bool {
    std::env::var_os(CLOCK_ENV).is_some()
        || estate_clock::config::gunnar_alias(CLOCK_ENV)
            .is_some_and(|alias| std::env::var_os(alias).is_some())
        || (appliance_pid1::net::is_init() && appliance_pid1::net::booted_by_korp_start())
}

/// Say what an attempt concluded, at the level it deserves.
fn log(report: &Report) {
    let summary = report.summary();
    match report.outcome {
        Outcome::Stepped | Outcome::Slewed | Outcome::AlreadyRight => {
            tracing::info!(
                outcome = report.outcome.as_str(),
                offset_ms = report.offset_ms,
                "clock: {summary}"
            )
        }
        Outcome::NotConfigured | Outcome::Off => {
            tracing::info!(outcome = report.outcome.as_str(), "clock: {summary}")
        }
        Outcome::Misconfigured | Outcome::NoQuorum | Outcome::Disagree | Outcome::ApplyFailed => {
            tracing::warn!(
                outcome = report.outcome.as_str(),
                "clock: {summary}. If this box's clock is wrong, every credential it is handed \
                 will be refused."
            )
        }
    }
    for row in &report.sources {
        match (&row.offset_ms, &row.error) {
            (_, Some(why)) => tracing::warn!(source = %row.source, %why, "clock source failed"),
            (Some(offset_ms), None) => {
                tracing::info!(source = %row.source, offset_ms, "clock source answered")
            }
            (None, None) => {}
        }
    }
}

/// One attempt on the blocking pool; the report is recorded by `sync` itself.
async fn attempt(phase: Trigger) -> Report {
    let config = crate::feed::clock_config();
    match tokio::task::spawn_blocking(move || sync(&config, phase)).await {
        Ok(report) => report,
        Err(err) => {
            let report = Report::without_sources(
                phase,
                Outcome::ApplyFailed,
                format!("the clock sync task did not complete: {err}"),
            );
            estate_clock::report::record(report.clone());
            report
        }
    }
}

/// **The boot call.** Sync once, stepping if the source says so.
pub async fn synchronise_at_boot() {
    if !owns_the_clock() {
        let report = Report::without_sources(
            Trigger::Boot,
            Outcome::Off,
            "not the appliance's PID 1, so the system clock is somebody else's to set",
        );
        tracing::debug!("clock: {}", report.summary());
        estate_clock::report::record(report);
        return;
    }
    log(&attempt(Trigger::Boot).await);
}

/// Whether a clock is configured at all (the medium's or the feed's).
fn configured() -> bool {
    !matches!(
        crate::feed::clock_config(),
        Ok(Config::NotConfigured) | Ok(Config::Off)
    )
}

/// **The running triggers**: a credential refused for skew, and the 6-hour
/// safety timer (`estate_clock::trigger::arm` has the rules and why there are
/// only these, and arms them once per process). Called after the boot sync, and
/// by a fed run before its first pause. Does nothing on a box that does not
/// own its clock, or whose clock is not configured — a clock the installer's
/// door feeds later arms them then ([`fed_arrived`]).
///
/// Both run on plain threads, never on the runtime: the skew handler is started
/// by the credential gate and must not borrow a worker for a TCP exchange, and a
/// sync is blocking by nature.
pub fn arm_triggers() {
    if !owns_the_clock() || !configured() {
        return;
    }
    estate_clock::trigger::arm(|trigger| {
        let config = crate::feed::clock_config();
        log(&sync(&config, trigger));
    });
}

/// One sync of the clock in force (the feed read now), logged.
fn fed_sync() -> Outcome {
    let config = crate::feed::clock_config();
    let report = sync(&config, Trigger::Timer);
    log(&report);
    report.outcome
}

/// This process's fed clock runs (`estate_clock::offer::FedRun`): the
/// triggers armed first, then syncs backing off while the front does not
/// answer; one run at a time, off the caller's thread; the tunnel coming up
/// cuts a pause short.
static FED: FedRun = FedRun::new(fed_sync, arm_triggers);

/// **The installer's door fed a clock source or key while this box runs**
/// (the pairing loop's first answer, `crate::feed`). Every sync already reads
/// the feed ([`crate::feed::clock_config`]), so this only makes the first one
/// happen NOW — a first boot whose medium carries no clock would otherwise
/// wait for the 6-hour timer, which was never armed — and arms the triggers.
///
/// ★ **Returns at once: the syncs run on their own thread** (cov-E r3
/// 2026-10-06, runs 2 and 4): called from the installer's FEED, a twin fed
/// before its tunnel was up once ran the whole backoff on the feed's thread
/// and for 219 s answered nobody. A clock re-fed meanwhile gets one more run
/// of its own when the one under way ends.
pub fn fed_arrived() {
    if !owns_the_clock() || !configured() {
        return;
    }
    if !FED.start() {
        tracing::warn!(
            "clock: the fed sync could not get a thread of its own; the triggers take over"
        );
    }
}

/// **This box's tunnel just came up** (the pairing loop's `TunnelUp`).
///
/// The fed clock source is the front's utility address, reached through
/// this tunnel on a box whose fed syncs ran before it was up (i54a run23:
/// the twin's three syncs timed out, then it paused 20 s while its tunnel
/// came up and the front's `BecomeTwin` was refused for skew seven times).
/// So: a clock not yet set right is asked NOW — the pause under way is cut
/// short, and a run is started if none is. Once per process; nothing when
/// the clock is already right.
pub fn tunnel_up() {
    if !owns_the_clock() || !configured() {
        return;
    }
    FED.tunnel_up(last_outcome());
}

#[cfg(test)]
mod airgap {
    use super::*;
    use estate_clock::config::Source;

    /// A loopback listener that would see any dial, and an https source aimed at it.
    fn trap() -> (std::net::TcpListener, Result<Config, String>) {
        let trap = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
        trap.set_nonblocking(true).unwrap();
        let url = format!("https://{}/", trap.local_addr().unwrap());
        (
            trap,
            Ok(Config::Sources {
                sources: vec![Source::Https(url)],
                peer_key: None,
            }),
        )
    }

    fn dialled(trap: &std::net::TcpListener) -> bool {
        std::thread::sleep(std::time::Duration::from_millis(100));
        trap.accept().is_ok()
    }

    /// ★ **AIRGAP: a default build asks no https:// clock source.** The source is refused by
    /// the feature's name, and the trap on loopback saw no connection.
    #[cfg(not(feature = "clock-fetch"))]
    #[test]
    fn a_default_build_refuses_an_https_clock_source_by_name_and_dials_nothing() {
        let (trap, config) = trap();
        let report = sync(&config, Trigger::Timer);
        let said = format!("{report:?}");
        assert!(
            said.contains("clock-fetch"),
            "the refusal names the feature: {said}"
        );
        assert!(
            !dialled(&trap),
            "a default build dialled the https clock source"
        );
    }

    /// With `clock-fetch` on, today's behaviour: the https source is asked.
    #[cfg(feature = "clock-fetch")]
    #[test]
    fn a_clock_fetch_build_asks_its_https_clock_source_as_before() {
        let (trap, config) = trap();
        let report = sync(&config, Trigger::Timer);
        assert!(
            dialled(&trap),
            "a clock-fetch build did not ask its https source: {report:?}"
        );
    }
}