Skip to main content

runlimit_postgres/
lib.rs

1//! Replica-safe `PostgreSQL` storage for Runlimit fixed-window policies.
2//!
3//! [`PostgresLimiter`] implements the same anchored fixed-window model as the
4//! in-memory backend: the first admitted check starts a window, and subsequent
5//! checks share that window until it expires. `PostgreSQL`'s clock is the sole
6//! time authority, so callers on different hosts do not need synchronized
7//! clocks. Atomic batches use set-based lock, preflight, and update phases, so
8//! their number of SQL phases does not grow with the batch size. Single checks
9//! use the same separate logical-key, counter-row, and capacity-shard lock
10//! phases so every statement that tests row absence starts after the logical
11//! key lock has been acquired.
12//! [`PostgresLimiter`] implements [`runlimit_core::Limiter`] for async generic
13//! adapters.
14//! The optional `serde` feature enables validated [`PostgresConfig`] loading
15//! and the corresponding `runlimit-core` policy and response metadata.
16//!
17//! # Installation
18//!
19//! Call [`PostgresLimiter::migrate`] before serving traffic. It can share
20//! `SQLx`'s `_sqlx_migrations` table with application migrations only when every
21//! participating [`Migrator`] configures
22//! [`Migrator::set_ignore_missing`] to `true`.
23//! A strict application migrator will reject Runlimit's migration versions as
24//! unknown. The first migration creates `runlimit_fixed_windows` in the
25//! connection's default schema.
26//!
27//! [`MIGRATOR`] is the raw bundled migrator with `SQLx`'s strict defaults. It is
28//! intended for an exclusively managed migration history and connection. It
29//! does not add [`PostgresLimiter::migrate`]'s protection against returning a
30//! possibly session-locked connection to a pool after an error or cancellation.
31//! Applications that retain a strict host migrator should vendor the bundled
32//! Runlimit SQL as application-owned migrations, in order, using
33//! [`CREATE_RUNLIMIT_FIXED_WINDOWS_SQL`] and
34//! [`SET_RUNLIMIT_FIXED_WINDOWS_FILLFACTOR_SQL`] followed by
35//! [`BOUND_RUNLIMIT_FIXED_WINDOW_CARDINALITY_SQL`] instead of invoking either
36//! Runlimit migrator against the shared history.
37//!
38//! # Storage capacity
39//!
40//! Every counter belongs to one of [`CAPACITY_SHARD_COUNT`] persistent shards.
41//! The migration-maintained ledger enforces
42//! [`HARD_MAX_ROWS_PER_SHARD`] even for older replicas, while
43//! [`PostgresConfig::maximum_rows_per_shard`] lets current replicas fail closed
44//! at a lower operational bound. A full shard denies only new storage keys;
45//! existing rows remain usable and active rows are never evicted. Expiry alone
46//! does not release a row. Capacity becomes reusable only after
47//! [`PostgresLimiter::cleanup_expired`] commits its deletion.
48//!
49//! # Failure semantics
50//!
51//! Pre-commit database failures and pre-commit timeouts mean the transaction
52//! did not commit. A
53//! [`CheckError::CommitOutcomeUnknown`] or [`CheckError::CommitTimedOut`] error
54//! means `PostgreSQL` may have committed even though the client did not receive
55//! confirmation. [`CheckError::CommittedResponseInvariant`] means the commit
56//! was confirmed but its response was unusable. Fail closed in all three cases,
57//! and do not blindly retry a check that may already have consumed quota.
58//! Once the database has produced a denial, an explicit rollback failure or
59//! deadline cannot replace that valid decision; the connection is discarded so
60//! closing it rolls back the non-mutating denied transaction.
61//! Dropping a check or cleanup future prevents the backend from reporting its
62//! outcome; if cancellation races with commit, callers must likewise treat the
63//! outcome as unknown. Prefer this backend's configured acquisition and
64//! operation budgets to a shorter outer timeout.
65//!
66//! The advisory-lock `v1` domain and derivation are a cross-replica protocol.
67//! They must not be changed in place: replicas from a rolling deployment must
68//! calculate identical lock IDs for the same persistent counter. `PostgreSQL`'s
69//! advisory-lock namespace is shared by every application in the same database.
70//! An unrelated or deliberately held matching `bigint` lock therefore blocks
71//! the affected Runlimit key until the operation deadline; use a trusted
72//! database boundary and restrict advisory-lock privileges for unrelated roles.
73//! The persisted capacity-shard derivation is likewise a cross-replica
74//! protocol: it XORs the leading bytes of the policy fingerprint and opaque
75//! subject key. Changing the derivation or shard count requires a new storage
76//! protocol and migration.
77//!
78//! # Integration tests
79//!
80//! The database tests are opt-in:
81//!
82//! ```text
83//! RUNLIMIT_POSTGRES_TEST_DATABASE_URL=postgresql:///runlimit_test \
84//! cargo test -p runlimit-postgres --test postgres -- --ignored
85//! ```
86
87use std::{
88    fmt,
89    future::Future,
90    sync::Arc,
91    time::{Duration, Instant as WallClockInstant},
92};
93
94use runlimit_core::{
95    AdmissionObservation, BatchDecision, Check, CleanupObservation, Decision, FixedWindowPolicy,
96    Limiter, Observation, Observer, observe_safely, validate_batch,
97};
98use sqlx::{
99    PgPool, Postgres,
100    migrate::{MigrateError, Migrator},
101    pool::PoolConnection,
102};
103use tokio::time::Instant;
104
105#[derive(Debug)]
106pub(crate) enum ConnectionOutcome<T> {
107    Reusable(T),
108    MustClose(T),
109}
110
111impl<T> ConnectionOutcome<T> {
112    fn map<U>(self, map: impl FnOnce(T) -> U) -> ConnectionOutcome<U> {
113        match self {
114            Self::Reusable(value) => ConnectionOutcome::Reusable(map(value)),
115            Self::MustClose(value) => ConnectionOutcome::MustClose(map(value)),
116        }
117    }
118}
119
120mod admission;
121mod config;
122mod errors;
123mod maintenance;
124mod protocol;
125
126pub use config::{PostgresConfig, PostgresConfigError};
127pub use errors::{CheckError, MaintenanceError};
128pub use protocol::{CAPACITY_SHARD_COUNT, HARD_MAX_ROWS_PER_SHARD};
129
130use admission::{
131    BatchSqlInput, acquire_check_connection, run_check_transaction, single_decision_from_batch,
132};
133use errors::check_error_consumption;
134use maintenance::{acquire_maintenance_connection, run_cleanup_transaction};
135
136#[cfg(test)]
137use admission::{
138    PendingAllowance, PendingBatchDenial, PendingDenial, PendingQuotaDenial, database_integer,
139    finish_denied_transaction,
140};
141#[cfg(test)]
142use protocol::{
143    BATCH_PREFLIGHT_SQL, BATCH_UPSERT_SQL, CLEANUP_SQL, advisory_lock_id, capacity_shard,
144};
145#[cfg(test)]
146use runlimit_core::BatchError;
147#[cfg(test)]
148use sqlx::postgres::types::PgInterval;
149
150/// SQL for the immutable `20260723000000_create_runlimit_fixed_windows`
151/// migration.
152///
153/// Strict host migrators can copy this SQL into their application-owned
154/// migration stream instead of running [`MIGRATOR`] against a shared
155/// `_sqlx_migrations` history. Apply it before
156/// [`SET_RUNLIMIT_FIXED_WINDOWS_FILLFACTOR_SQL`].
157pub const CREATE_RUNLIMIT_FIXED_WINDOWS_SQL: &str =
158    include_str!("../migrations/20260723000000_create_runlimit_fixed_windows.sql");
159
160/// SQL for the additive
161/// `20260725000000_set_runlimit_fixed_windows_fillfactor` migration.
162///
163/// Strict host migrators can copy this SQL into their application-owned
164/// migration stream after [`CREATE_RUNLIMIT_FIXED_WINDOWS_SQL`]. The
165/// application remains responsible for assigning versions that fit its own
166/// migration history.
167pub const SET_RUNLIMIT_FIXED_WINDOWS_FILLFACTOR_SQL: &str =
168    include_str!("../migrations/20260725000000_set_runlimit_fixed_windows_fillfactor.sql");
169
170/// SQL for the additive
171/// `20260726000000_bound_runlimit_fixed_window_cardinality` migration.
172///
173/// This migration assigns every persisted counter to one of 256 stable
174/// capacity shards and installs a trigger-maintained row-count ledger. Apply it
175/// after [`SET_RUNLIMIT_FIXED_WINDOWS_FILLFACTOR_SQL`].
176pub const BOUND_RUNLIMIT_FIXED_WINDOW_CARDINALITY_SQL: &str =
177    include_str!("../migrations/20260726000000_bound_runlimit_fixed_window_cardinality.sql");
178
179/// Raw bundled migrations required by [`PostgresLimiter`].
180///
181/// This value retains `SQLx`'s strict missing-version behavior and is suitable
182/// only for an exclusively managed migration history and connection. For a
183/// pooled application that shares `_sqlx_migrations`, prefer
184/// [`PostgresLimiter::migrate`] and configure every other participating
185/// [`Migrator`] to set [`Migrator::set_ignore_missing`] to `true`.
186pub static MIGRATOR: Migrator = sqlx::migrate!("./migrations");
187
188/// A replica-safe rate limiter backed by a `PostgreSQL` pool.
189///
190/// Cloning this value is cheap because [`PgPool`] is internally reference
191/// counted.
192#[derive(Clone)]
193pub struct PostgresLimiter {
194    pool: PgPool,
195    config: PostgresConfig,
196    observer: Option<Arc<dyn Observer>>,
197}
198
199impl fmt::Debug for PostgresLimiter {
200    fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
201        formatter
202            .debug_struct("PostgresLimiter")
203            .field("pool", &self.pool)
204            .field("config", &self.config)
205            .field("has_observer", &self.observer.is_some())
206            .finish_non_exhaustive()
207    }
208}
209
210impl PostgresLimiter {
211    /// Creates a limiter using `pool`.
212    ///
213    /// Call [`Self::migrate`] during application startup before admitting
214    /// requests.
215    pub const fn new(pool: PgPool) -> Self {
216        Self {
217            pool,
218            config: PostgresConfig::new(),
219            observer: None,
220        }
221    }
222
223    /// Creates a limiter using an explicit runtime configuration.
224    pub const fn with_config(pool: PgPool, config: PostgresConfig) -> Self {
225        Self {
226            pool,
227            config,
228            observer: None,
229        }
230    }
231
232    /// Returns this limiter with an operational observer.
233    ///
234    /// Observer callbacks run only after a database operation has released its
235    /// connection. Callback panics are isolated and cannot change admission or
236    /// cleanup results.
237    #[must_use]
238    pub fn with_observer(mut self, observer: Arc<dyn Observer>) -> Self {
239        self.observer = Some(observer);
240        self
241    }
242
243    /// Returns the underlying connection pool.
244    pub const fn pool(&self) -> &PgPool {
245        &self.pool
246    }
247
248    /// Returns the backend runtime configuration.
249    pub const fn config(&self) -> PostgresConfig {
250        self.config
251    }
252
253    /// Applies the bundled Runlimit database migrations.
254    ///
255    /// Unrelated successful versions in `_sqlx_migrations` are ignored, which
256    /// permits a shared migration history only when every other participating
257    /// `SQLx` migrator also sets [`Migrator::set_ignore_missing`] to `true`.
258    /// Known Runlimit versions still receive `SQLx`'s checksum validation.
259    /// Applications that keep a strict host migrator should instead vendor the
260    /// bundled SQL into their application-owned migration stream.
261    ///
262    /// Migration locking is session scoped. An acquired connection is returned
263    /// to the pool only after a successful run and unlock; an error or
264    /// cancellation closes it instead.
265    ///
266    /// # Errors
267    ///
268    /// Returns an error if migration metadata cannot be read or `PostgreSQL`
269    /// rejects a migration.
270    pub async fn migrate(&self) -> Result<(), MigrateError> {
271        let connection = self.pool.acquire().await?;
272        let mut guarded_connection = ConnectionCancellationGuard::new(connection);
273        let mut migrator = sqlx::migrate!("./migrations");
274        migrator.set_ignore_missing(true);
275
276        let result = migrator
277            .run_direct(&mut **guarded_connection.connection())
278            .await;
279        if result.is_ok() {
280            guarded_connection.reuse();
281        }
282        result
283    }
284
285    /// Checks and, when allowed, consumes one fixed-window quota.
286    ///
287    /// A denied decision does not change the stored counter.
288    ///
289    /// # Errors
290    ///
291    /// Returns [`CheckError::DefinitelyNotConsumed`] for failures before a
292    /// successful commit and [`CheckError::CommitOutcomeUnknown`] when commit
293    /// confirmation is unavailable. An internal single-response invariant
294    /// failure reports whether the batch was rolled back or committed.
295    pub async fn check(&self, check: &Check<'_>) -> Result<Decision, CheckError> {
296        let started = WallClockInstant::now();
297        let result = self
298            .check_all_unobserved(std::slice::from_ref(check))
299            .await
300            .and_then(single_decision_from_batch);
301        self.observe_single_admission(check, &result, started.elapsed());
302        result
303    }
304
305    /// Atomically checks and consumes every quota in `checks`.
306    ///
307    /// Rows are acquired in a deterministic storage-key order to avoid
308    /// deadlocks between overlapping batches. Returned allowed decisions retain
309    /// the caller's input order. Repeated occurrences of the same storage key
310    /// are rejected before a connection is acquired because their independent
311    /// decision metadata would be ambiguous.
312    ///
313    /// New keys lock their capacity-ledger rows in shard order. If the earliest
314    /// caller-ordered failure is the configured shard bound, the batch returns
315    /// [`runlimit_core::Denial::storage_capacity`] with no retry duration and changes neither
316    /// quota nor storage. Existing and expired target rows need no new slot.
317    ///
318    /// If any check is denied, the backend attempts an explicit rollback and
319    /// the returned denial index is the earliest denied item in caller order.
320    /// A rollback error or deadline does not replace the denial because no
321    /// counter was changed; instead, the connection is discarded so socket
322    /// closure rolls the transaction back. Passing an empty slice succeeds
323    /// without acquiring a database connection.
324    ///
325    /// # Errors
326    ///
327    /// Pool-acquisition or operation timeout before commit is
328    /// [`CheckError::TimedOutBeforeCommit`]. Other database failures before
329    /// commit are [`CheckError::DefinitelyNotConsumed`]. Any commit error is
330    /// conservatively classified as [`CheckError::CommitOutcomeUnknown`].
331    pub async fn check_all(&self, checks: &[Check<'_>]) -> Result<BatchDecision, CheckError> {
332        let started = WallClockInstant::now();
333        let result = self.check_all_unobserved(checks).await;
334        self.observe_batch_admission(checks, &result, started.elapsed());
335        result
336    }
337
338    async fn check_all_unobserved(
339        &self,
340        checks: &[Check<'_>],
341    ) -> Result<BatchDecision, CheckError> {
342        validate_batch(checks, self.config.max_batch_size())?;
343        if checks.is_empty() {
344            return Ok(BatchDecision::allowed(Vec::new()));
345        }
346
347        // Materialize every SQL array and both lock orders before reserving a
348        // pool slot. Every database phase then consumes the exact same key
349        // arrays, preventing identity drift between locking and mutation.
350        let input = BatchSqlInput::from_checks(checks);
351        let connection =
352            acquire_check_connection(&self.pool, self.config.pool_acquire_timeout()).await?;
353        // Pool contention does not consume the database-work budget. A
354        // survivor receives a fresh deadline for begin, statements, locks,
355        // rollback, and commit.
356        let deadline = Instant::now() + self.config.operation_timeout();
357        let mut guarded_connection = ConnectionCancellationGuard::new(connection);
358        let outcome = run_check_transaction(
359            guarded_connection.connection(),
360            &input,
361            checks,
362            self.config.maximum_rows_per_shard(),
363            deadline,
364        )
365        .await;
366        guarded_connection.finish(outcome)
367    }
368
369    /// Deletes at most `maximum_rows` expired windows and releases their
370    /// capacity-ledger slots in the same transaction.
371    ///
372    /// Cleanup uses `FOR UPDATE SKIP LOCKED`, so it does not wait behind active
373    /// checks and cannot remove a row concurrently being renewed. A zero limit
374    /// performs no database work.
375    ///
376    /// # Errors
377    ///
378    /// Returns an error if `PostgreSQL` rejects the maintenance transaction or
379    /// if either [`PostgresConfig::pool_acquire_timeout`] or the fresh
380    /// [`PostgresConfig::operation_timeout`] budget is exceeded. A commit error
381    /// or commit timeout means rows may have been removed; inspect
382    /// [`MaintenanceError::may_have_removed_rows`]. Cleanup only targets
383    /// already-expired rows and never consumes quota.
384    pub async fn cleanup_expired(&self, maximum_rows: u32) -> Result<u64, MaintenanceError> {
385        let started = WallClockInstant::now();
386        let result = self.cleanup_expired_unobserved(maximum_rows).await;
387        self.observe_cleanup(maximum_rows, &result, started.elapsed());
388        result
389    }
390
391    async fn cleanup_expired_unobserved(&self, maximum_rows: u32) -> Result<u64, MaintenanceError> {
392        if maximum_rows == 0 {
393            return Ok(0);
394        }
395
396        let connection =
397            acquire_maintenance_connection(&self.pool, self.config.pool_acquire_timeout()).await?;
398        let deadline = Instant::now() + self.config.operation_timeout();
399        let mut guarded_connection = ConnectionCancellationGuard::new(connection);
400        let outcome =
401            run_cleanup_transaction(guarded_connection.connection(), maximum_rows, deadline).await;
402        guarded_connection.finish(outcome)
403    }
404
405    fn observe_single_admission(
406        &self,
407        check: &Check<'_>,
408        result: &Result<Decision, CheckError>,
409        elapsed: Duration,
410    ) {
411        match result {
412            Ok(decision) => self
413                .observe_admission(|| AdmissionObservation::from_check(check, decision, elapsed)),
414            Err(error) => self.observe_admission(|| {
415                AdmissionObservation::failed_check(check, check_error_consumption(error), elapsed)
416            }),
417        }
418    }
419
420    fn observe_batch_admission(
421        &self,
422        checks: &[Check<'_>],
423        result: &Result<BatchDecision, CheckError>,
424        elapsed: Duration,
425    ) {
426        match result {
427            Ok(decision) => self
428                .observe_admission(|| AdmissionObservation::from_batch(checks, decision, elapsed)),
429            Err(error) => {
430                self.observe_admission(|| {
431                    if let [check] = checks {
432                        AdmissionObservation::failed_batch_for_check(
433                            check,
434                            check_error_consumption(error),
435                            elapsed,
436                        )
437                    } else {
438                        AdmissionObservation::failed_batch(
439                            checks.len(),
440                            check_error_consumption(error),
441                            elapsed,
442                        )
443                    }
444                });
445            }
446        }
447    }
448
449    fn observe_admission<'a>(&self, build_observation: impl FnOnce() -> AdmissionObservation<'a>) {
450        let Some(observer) = &self.observer else {
451            return;
452        };
453        let admission = build_observation();
454        observe_safely(observer.as_ref(), &Observation::Admission(admission));
455    }
456
457    fn observe_cleanup(
458        &self,
459        requested: u32,
460        result: &Result<u64, MaintenanceError>,
461        elapsed: Duration,
462    ) {
463        let Some(observer) = &self.observer else {
464            return;
465        };
466        let requested = usize::try_from(requested).unwrap_or(usize::MAX);
467        let observation = match result {
468            Ok(removed) => CleanupObservation::confirmed(requested, *removed, elapsed),
469            Err(error) if error.may_have_removed_rows() => {
470                CleanupObservation::outcome_unknown(requested, elapsed)
471            }
472            Err(_) => CleanupObservation::definitely_no_effect(requested, elapsed),
473        };
474        observe_safely(observer.as_ref(), &Observation::Cleanup(observation));
475    }
476}
477
478impl Limiter for PostgresLimiter {
479    type Policy = FixedWindowPolicy;
480    type Error = CheckError;
481
482    fn check(
483        &self,
484        check: &Check<'_>,
485    ) -> impl Future<Output = Result<Decision, Self::Error>> + Send {
486        PostgresLimiter::check(self, check)
487    }
488
489    fn check_all(
490        &self,
491        checks: &[Check<'_>],
492    ) -> impl Future<Output = Result<BatchDecision, Self::Error>> + Send {
493        PostgresLimiter::check_all(self, checks)
494    }
495}
496
497struct ConnectionCancellationGuard {
498    connection: PoolConnection<Postgres>,
499    armed: bool,
500}
501
502impl ConnectionCancellationGuard {
503    const fn new(connection: PoolConnection<Postgres>) -> Self {
504        Self {
505            connection,
506            armed: true,
507        }
508    }
509
510    const fn connection(&mut self) -> &mut PoolConnection<Postgres> {
511        &mut self.connection
512    }
513
514    const fn reuse(&mut self) {
515        self.armed = false;
516    }
517
518    fn finish<T, E>(mut self, outcome: ConnectionOutcome<Result<T, E>>) -> Result<T, E> {
519        match outcome {
520            ConnectionOutcome::Reusable(result) => {
521                self.reuse();
522                result
523            }
524            ConnectionOutcome::MustClose(result) => result,
525        }
526    }
527}
528
529impl Drop for ConnectionCancellationGuard {
530    fn drop(&mut self) {
531        if self.armed {
532            // Dropping an SQLx query future does not send a PostgreSQL cancel
533            // request. Close the socket instead of returning a potentially
534            // busy connection to the pool when either Runlimit's deadline or
535            // cancellation of the outer limiter future wins the race.
536            self.connection.close_on_drop();
537        }
538    }
539}
540
541#[cfg(test)]
542mod tests {
543    use std::{
544        sync::{Arc, Mutex},
545        time::Duration,
546    };
547
548    use super::*;
549    use runlimit_core::{
550        AdmissionOperation, AdmissionOutcome, ConsumptionStatus, FixedWindowPolicy, MAX_LIMIT,
551        MAX_WINDOW, MAX_WINDOW_MILLIS, PolicyId, QuotaDenial, ScopeId, SubjectKey,
552    };
553
554    fn policy(id: &str, scope: &str) -> FixedWindowPolicy {
555        FixedWindowPolicy::new(
556            PolicyId::new(id).expect("valid policy identifier"),
557            ScopeId::new(scope).expect("valid scope identifier"),
558            10,
559            Duration::from_secs(60),
560        )
561        .expect("valid policy")
562    }
563
564    fn key(byte: u8) -> SubjectKey {
565        SubjectKey::from_digest([byte; 32])
566    }
567
568    #[derive(Default)]
569    struct RecordingObserver {
570        events: Mutex<Vec<(AdmissionOperation, AdmissionOutcome, ConsumptionStatus)>>,
571        cleanups: Mutex<Vec<(usize, Option<u64>, ConsumptionStatus)>>,
572    }
573
574    impl Observer for RecordingObserver {
575        fn observe(&self, observation: &Observation<'_>) {
576            match observation {
577                Observation::Admission(admission) => self.events.lock().unwrap().push((
578                    admission.operation(),
579                    admission.outcome(),
580                    admission.consumption(),
581                )),
582                Observation::Cleanup(cleanup) => self.cleanups.lock().unwrap().push((
583                    cleanup.requested(),
584                    cleanup.removed(),
585                    cleanup.consumption(),
586                )),
587                _ => {}
588            }
589        }
590    }
591
592    #[derive(Clone, Copy, Debug, Eq, PartialEq)]
593    struct RecordedAdmissionMetadata {
594        operation: AdmissionOperation,
595        batch_size: usize,
596        has_policy_id: bool,
597        has_scope_id: bool,
598        has_policy_fingerprint: bool,
599        consumption: ConsumptionStatus,
600    }
601
602    #[derive(Default)]
603    struct AdmissionMetadataObserver {
604        admissions: Mutex<Vec<RecordedAdmissionMetadata>>,
605    }
606
607    impl Observer for AdmissionMetadataObserver {
608        fn observe(&self, observation: &Observation<'_>) {
609            let Observation::Admission(admission) = observation else {
610                return;
611            };
612            self.admissions
613                .lock()
614                .unwrap()
615                .push(RecordedAdmissionMetadata {
616                    operation: admission.operation(),
617                    batch_size: admission.batch_size(),
618                    has_policy_id: admission.policy_id().is_some(),
619                    has_scope_id: admission.scope_id().is_some(),
620                    has_policy_fingerprint: admission.policy_fingerprint().is_some(),
621                    consumption: admission.consumption(),
622                });
623        }
624    }
625
626    struct PanickingObserver;
627
628    impl Observer for PanickingObserver {
629        fn observe(&self, _: &Observation<'_>) {
630            panic!("injected observer panic");
631        }
632    }
633
634    #[test]
635    fn portable_core_boundaries_fit_database_values_exactly() {
636        let policy = FixedWindowPolicy::new(
637            PolicyId::new("portable.maximum").unwrap(),
638            ScopeId::new("subject").unwrap(),
639            MAX_LIMIT,
640            MAX_WINDOW,
641        )
642        .unwrap();
643        let check = Check::with_cost(&policy, key(1), MAX_LIMIT).unwrap();
644
645        assert_eq!(database_integer(policy.limit()), i64::MAX);
646        assert_eq!(database_integer(check.cost()), i64::MAX);
647        let interval = PgInterval::try_from(policy.window()).unwrap();
648        assert_eq!(interval.months, 0);
649        assert_eq!(interval.days, 0);
650        assert_eq!(
651            interval.microseconds,
652            i64::try_from(MAX_WINDOW_MILLIS * 1_000).unwrap()
653        );
654    }
655
656    #[test]
657    fn row_lock_permutation_orders_complete_counter_keys() {
658        let alpha_client = policy("alpha", "client");
659        let alpha_identity = policy("alpha", "identity");
660        let beta_client = policy("beta", "client");
661        let checks = [
662            Check::new(&beta_client, key(0)),
663            Check::new(&alpha_identity, key(0)),
664            Check::new(&alpha_client, key(1)),
665            Check::new(&alpha_client, key(0)),
666        ];
667
668        let input = BatchSqlInput::from_checks(&checks);
669        let locked_keys = input
670            .lock_input_positions
671            .iter()
672            .map(|position| checks[usize::try_from(*position - 1).unwrap()].counter_key())
673            .collect::<Vec<_>>();
674
675        assert!(locked_keys.windows(2).all(|pair| pair[0] < pair[1]));
676    }
677
678    #[test]
679    fn advisory_lock_protocol_has_a_golden_vector() {
680        let policy = policy("login", "client");
681        let check = Check::new(&policy, key(7));
682
683        assert_eq!(
684            advisory_lock_id(check.counter_key()),
685            4_549_358_434_381_602_087
686        );
687    }
688
689    #[test]
690    fn capacity_shard_protocol_has_a_golden_vector() {
691        let policy = policy("login", "client");
692        let check = Check::new(&policy, key(7));
693
694        assert_eq!(capacity_shard(check.counter_key()), 141);
695    }
696
697    #[tokio::test]
698    async fn observer_reports_connection_free_operations_and_isolates_panics() {
699        let pool = sqlx::postgres::PgPoolOptions::new()
700            .connect_lazy("postgresql://runlimit:runlimit@127.0.0.1:1/runlimit")
701            .expect("syntactically valid database URL");
702        let observer = Arc::new(RecordingObserver::default());
703        let limiter = PostgresLimiter::new(pool.clone()).with_observer(observer.clone());
704
705        assert_eq!(
706            limiter.check_all(&[]).await.unwrap(),
707            BatchDecision::allowed(Vec::new())
708        );
709        assert_eq!(limiter.cleanup_expired(0).await.unwrap(), 0);
710        assert_eq!(
711            observer.events.lock().unwrap().as_slice(),
712            [(
713                AdmissionOperation::Batch,
714                AdmissionOutcome::Allowed,
715                ConsumptionStatus::NotConsumed,
716            )]
717        );
718        assert_eq!(
719            observer.cleanups.lock().unwrap().as_slice(),
720            [(0, Some(0), ConsumptionStatus::Consumed)]
721        );
722
723        let panicking = PostgresLimiter::new(pool).with_observer(Arc::new(PanickingObserver));
724        assert_eq!(
725            panicking.check_all(&[]).await.unwrap(),
726            BatchDecision::allowed(Vec::new())
727        );
728        assert_eq!(panicking.cleanup_expired(0).await.unwrap(), 0);
729    }
730
731    #[tokio::test]
732    async fn failed_postgres_batches_preserve_one_item_policy_metadata() {
733        let pool = sqlx::postgres::PgPoolOptions::new()
734            .connect_lazy("postgresql://runlimit:runlimit@127.0.0.1:1/runlimit")
735            .expect("syntactically valid database URL");
736        let observer = Arc::new(AdmissionMetadataObserver::default());
737        let limiter = PostgresLimiter::new(pool).with_observer(observer.clone());
738        let first_policy = policy("observed.failure.first", "client");
739        let second_policy = policy("observed.failure.second", "client");
740        let first = Check::new(&first_policy, key(1));
741        let second = Check::new(&second_policy, key(2));
742        let elapsed = Duration::from_millis(5);
743
744        limiter.observe_single_admission(&first, &Err(CheckError::CommitTimedOut), elapsed);
745        limiter.observe_batch_admission(&[first], &Err(CheckError::CommitTimedOut), elapsed);
746        limiter.observe_batch_admission(
747            &[first, second],
748            &Err(CheckError::CommitTimedOut),
749            elapsed,
750        );
751
752        assert_eq!(
753            observer.admissions.lock().unwrap().as_slice(),
754            [
755                RecordedAdmissionMetadata {
756                    operation: AdmissionOperation::Check,
757                    batch_size: 1,
758                    has_policy_id: true,
759                    has_scope_id: true,
760                    has_policy_fingerprint: true,
761                    consumption: ConsumptionStatus::PossiblyConsumed,
762                },
763                RecordedAdmissionMetadata {
764                    operation: AdmissionOperation::Batch,
765                    batch_size: 1,
766                    has_policy_id: true,
767                    has_scope_id: true,
768                    has_policy_fingerprint: true,
769                    consumption: ConsumptionStatus::PossiblyConsumed,
770                },
771                RecordedAdmissionMetadata {
772                    operation: AdmissionOperation::Batch,
773                    batch_size: 2,
774                    has_policy_id: false,
775                    has_scope_id: false,
776                    has_policy_fingerprint: false,
777                    consumption: ConsumptionStatus::PossiblyConsumed,
778                },
779            ]
780        );
781    }
782
783    #[tokio::test]
784    async fn cleanup_observations_map_every_backend_effect_class() {
785        let pool = sqlx::postgres::PgPoolOptions::new()
786            .connect_lazy("postgresql://runlimit:runlimit@127.0.0.1:1/runlimit")
787            .expect("syntactically valid database URL");
788        let observer = Arc::new(RecordingObserver::default());
789        let limiter = PostgresLimiter::new(pool).with_observer(observer.clone());
790        let elapsed = Duration::from_millis(5);
791
792        limiter.observe_cleanup(7, &Ok(3), elapsed);
793        limiter.observe_cleanup(
794            7,
795            &Err(MaintenanceError::Database(sqlx::Error::RowNotFound)),
796            elapsed,
797        );
798        limiter.observe_cleanup(
799            7,
800            &Err(MaintenanceError::TimedOutBeforeCommit {
801                operation: "testing",
802            }),
803            elapsed,
804        );
805        limiter.observe_cleanup(
806            7,
807            &Err(MaintenanceError::CommitOutcomeUnknown(
808                sqlx::Error::RowNotFound,
809            )),
810            elapsed,
811        );
812        limiter.observe_cleanup(7, &Err(MaintenanceError::CommitTimedOut), elapsed);
813
814        assert_eq!(
815            observer.cleanups.lock().unwrap().as_slice(),
816            [
817                (7, Some(3), ConsumptionStatus::Consumed),
818                (7, Some(0), ConsumptionStatus::NotConsumed),
819                (7, Some(0), ConsumptionStatus::NotConsumed),
820                (7, None, ConsumptionStatus::PossiblyConsumed),
821                (7, None, ConsumptionStatus::PossiblyConsumed),
822            ]
823        );
824    }
825
826    #[test]
827    fn response_metadata_subtracts_only_authoritative_elapsed_time() {
828        let allowed = PendingAllowance {
829            limit: 10,
830            remaining: 4,
831            reset_from_sample: Duration::from_millis(250),
832        }
833        .finish(Duration::from_millis(80));
834        let denied = PendingDenial::Quota(PendingQuotaDenial {
835            limit: 10,
836            retry_from_sample: Duration::from_millis(150),
837        })
838        .finish(Duration::from_millis(80));
839
840        assert_eq!(
841            allowed.replenishes_after(),
842            Some(Duration::from_millis(170))
843        );
844        assert_eq!(denied.retry_after(), Some(Duration::from_millis(70)));
845        assert_eq!(denied.retry_after_seconds(), Some(1));
846    }
847
848    #[tokio::test]
849    async fn denied_decision_survives_rollback_failure_and_discards_connection() {
850        let outcome = finish_denied_transaction(
851            Instant::now() + Duration::from_secs(1),
852            PendingBatchDenial::Enforced {
853                index: 2,
854                denial: PendingDenial::Quota(PendingQuotaDenial {
855                    limit: 10,
856                    retry_from_sample: Duration::from_millis(150),
857                }),
858            },
859            Duration::from_millis(20),
860            std::future::ready(Err(sqlx::Error::Protocol(
861                "injected rollback failure".to_owned(),
862            ))),
863        )
864        .await;
865
866        let ConnectionOutcome::MustClose(decision) = outcome else {
867            panic!("rollback failure must close the connection")
868        };
869        assert_eq!(decision.denied_index(), Some(2));
870        assert_eq!(
871            decision.quota_denial(),
872            Some(QuotaDenial::try_new(10, Duration::from_millis(130)).unwrap())
873        );
874    }
875
876    #[tokio::test]
877    async fn denied_decision_survives_rollback_deadline_and_discards_connection() {
878        let outcome = finish_denied_transaction(
879            Instant::now(),
880            PendingBatchDenial::Enforced {
881                index: 0,
882                denial: PendingDenial::Quota(PendingQuotaDenial {
883                    limit: 1,
884                    retry_from_sample: Duration::from_millis(50),
885                }),
886            },
887            Duration::ZERO,
888            std::future::pending(),
889        )
890        .await;
891
892        let ConnectionOutcome::MustClose(decision) = outcome else {
893            panic!("rollback deadline must close the connection")
894        };
895        assert_eq!(decision.denied_index(), Some(0));
896        assert_eq!(decision.quota_denial().map(QuotaDenial::capacity), Some(1));
897    }
898
899    #[tokio::test]
900    async fn shadow_quota_denial_is_reported_after_rollback() {
901        let outcome = finish_denied_transaction(
902            Instant::now() + Duration::from_secs(1),
903            PendingBatchDenial::Shadow {
904                index: 0,
905                denial: PendingQuotaDenial {
906                    limit: 1,
907                    retry_from_sample: Duration::from_secs(1),
908                },
909            },
910            Duration::ZERO,
911            std::future::ready(Ok(())),
912        )
913        .await;
914
915        let ConnectionOutcome::Reusable(decision) = outcome else {
916            panic!("successful rollback must reuse the connection")
917        };
918        assert!(decision.is_shadow_denied());
919        assert_eq!(decision.denied_index(), Some(0));
920        assert_eq!(decision.quota_denial().map(QuotaDenial::capacity), Some(1));
921    }
922
923    #[test]
924    fn configuration_bounds_batches_and_deadlines() {
925        let defaults = PostgresConfig::new();
926        assert_eq!(defaults.maximum_rows_per_shard(), 4_096);
927        assert_eq!(defaults.max_batch_size(), 32);
928        assert_eq!(defaults.pool_acquire_timeout(), Duration::from_secs(3));
929        assert_eq!(defaults.operation_timeout(), Duration::from_secs(3));
930        assert_eq!(
931            defaults.with_maximum_rows_per_shard(0),
932            Err(PostgresConfigError::ZeroMaximumRowsPerShard)
933        );
934        assert_eq!(
935            defaults.with_maximum_rows_per_shard(HARD_MAX_ROWS_PER_SHARD + 1),
936            Err(PostgresConfigError::MaximumRowsPerShardTooLarge {
937                actual: HARD_MAX_ROWS_PER_SHARD + 1,
938                maximum: HARD_MAX_ROWS_PER_SHARD,
939            })
940        );
941        assert_eq!(
942            defaults.with_max_batch_size(0),
943            Err(PostgresConfigError::ZeroBatchSize)
944        );
945        assert_eq!(
946            defaults.with_pool_acquire_timeout(Duration::ZERO),
947            Err(PostgresConfigError::ZeroPoolAcquireTimeout)
948        );
949        assert_eq!(
950            defaults.with_pool_acquire_timeout(Duration::from_secs(61)),
951            Err(PostgresConfigError::PoolAcquireTimeoutTooLong {
952                actual: Duration::from_secs(61),
953                maximum: Duration::from_secs(60),
954            })
955        );
956        assert_eq!(
957            defaults.with_operation_timeout(Duration::ZERO),
958            Err(PostgresConfigError::ZeroOperationTimeout)
959        );
960        assert_eq!(
961            defaults.with_operation_timeout(Duration::from_secs(61)),
962            Err(PostgresConfigError::OperationTimeoutTooLong {
963                actual: Duration::from_secs(61),
964                maximum: Duration::from_secs(60),
965            })
966        );
967    }
968
969    #[tokio::test]
970    async fn duplicate_storage_keys_fail_before_connecting() {
971        let alpha = policy("alpha", "client");
972        let beta = policy("beta", "client");
973        let checks = [
974            Check::new(&beta, key(7)),
975            Check::new(&beta, key(7)),
976            Check::new(&alpha, key(8)),
977            Check::new(&alpha, key(8)),
978        ];
979        let pool = sqlx::postgres::PgPoolOptions::new()
980            .connect_lazy("postgresql://runlimit:runlimit@127.0.0.1:1/runlimit")
981            .expect("syntactically valid database URL");
982        let limiter = PostgresLimiter::new(pool);
983
984        let error = limiter
985            .check_all(&checks)
986            .await
987            .expect_err("duplicate keys must be rejected");
988
989        assert!(matches!(
990            error,
991            CheckError::InvalidBatch(BatchError::DuplicateKey {
992                first_index: 0,
993                duplicate_index: 1,
994            })
995        ));
996    }
997
998    #[tokio::test]
999    async fn oversized_batch_fails_before_connecting() {
1000        let first_policy = policy("login", "client");
1001        let second_policy = policy("signup", "client");
1002        let checks = [
1003            Check::new(&first_policy, key(1)),
1004            Check::new(&second_policy, key(2)),
1005        ];
1006        let pool = sqlx::postgres::PgPoolOptions::new()
1007            .connect_lazy("postgresql://runlimit:runlimit@127.0.0.1:1/runlimit")
1008            .expect("syntactically valid database URL");
1009        let config = PostgresConfig::new()
1010            .with_max_batch_size(1)
1011            .expect("one is a valid batch limit");
1012        let limiter = PostgresLimiter::with_config(pool, config);
1013
1014        let error = limiter
1015            .check_all(&checks)
1016            .await
1017            .expect_err("oversized batch must be rejected");
1018
1019        assert!(matches!(
1020            error,
1021            CheckError::InvalidBatch(BatchError::BatchTooLarge {
1022                actual: 2,
1023                maximum: 1,
1024            })
1025        ));
1026    }
1027
1028    #[test]
1029    fn error_reports_consumption_uncertainty() {
1030        let definite = CheckError::DefinitelyNotConsumed(sqlx::Error::RowNotFound);
1031        let unknown = CheckError::CommitOutcomeUnknown(sqlx::Error::RowNotFound);
1032        let rolled_back_invariant = CheckError::ResponseInvariant;
1033        let committed_invariant = CheckError::CommittedResponseInvariant;
1034        let cleanup_definite = MaintenanceError::Database(sqlx::Error::RowNotFound);
1035        let cleanup_unknown = MaintenanceError::CommitOutcomeUnknown(sqlx::Error::RowNotFound);
1036
1037        assert!(!definite.may_have_consumed_quota());
1038        assert!(unknown.may_have_consumed_quota());
1039        assert!(CheckError::CommitTimedOut.may_have_consumed_quota());
1040        assert!(!rolled_back_invariant.may_have_consumed_quota());
1041        assert!(committed_invariant.may_have_consumed_quota());
1042        assert_eq!(
1043            check_error_consumption(&definite),
1044            ConsumptionStatus::NotConsumed
1045        );
1046        assert_eq!(
1047            check_error_consumption(&unknown),
1048            ConsumptionStatus::PossiblyConsumed
1049        );
1050        assert_eq!(
1051            check_error_consumption(&CheckError::CommitTimedOut),
1052            ConsumptionStatus::PossiblyConsumed
1053        );
1054        assert_eq!(
1055            check_error_consumption(&rolled_back_invariant),
1056            ConsumptionStatus::NotConsumed
1057        );
1058        assert_eq!(
1059            check_error_consumption(&committed_invariant),
1060            ConsumptionStatus::Consumed
1061        );
1062        assert!(!cleanup_definite.may_have_removed_rows());
1063        assert!(cleanup_unknown.may_have_removed_rows());
1064        assert!(MaintenanceError::CommitTimedOut.may_have_removed_rows());
1065    }
1066
1067    #[test]
1068    fn malformed_single_responses_preserve_consumption_status() {
1069        let denial = QuotaDenial::try_new(1, Duration::from_secs(1)).unwrap();
1070
1071        let committed = single_decision_from_batch(BatchDecision::allowed(Vec::new()))
1072            .expect_err("malformed allowed response follows a confirmed commit");
1073        let rolled_back = single_decision_from_batch(BatchDecision::denied(1, denial))
1074            .expect_err("malformed denied response follows a rollback");
1075
1076        assert!(matches!(committed, CheckError::CommittedResponseInvariant));
1077        assert!(committed.may_have_consumed_quota());
1078        assert!(matches!(rolled_back, CheckError::ResponseInvariant));
1079        assert!(!rolled_back.may_have_consumed_quota());
1080    }
1081
1082    #[test]
1083    fn cleanup_query_is_bounded_and_nonblocking() {
1084        assert_eq!(
1085            CLEANUP_SQL.matches("pg_catalog.clock_timestamp()").count(),
1086            1
1087        );
1088        assert!(CLEANUP_SQL.contains("AS MATERIALIZED"));
1089        assert!(CLEANUP_SQL.contains("SELECT sampled_at\n        FROM authoritative_time"));
1090        assert!(!CLEANUP_SQL.contains("window_expires_at <= pg_catalog.clock_timestamp()"));
1091        assert!(CLEANUP_SQL.contains("LIMIT $1"));
1092        assert!(CLEANUP_SQL.contains("FOR UPDATE SKIP LOCKED"));
1093        assert!(CLEANUP_SQL.contains("ORDER BY capacity.capacity_shard"));
1094        assert!(CLEANUP_SQL.contains("FOR UPDATE OF capacity"));
1095    }
1096
1097    #[test]
1098    fn every_production_clock_call_is_catalog_qualified() {
1099        for (name, sql) in [
1100            ("batch preflight", BATCH_PREFLIGHT_SQL),
1101            ("batch upsert", BATCH_UPSERT_SQL),
1102            ("cleanup", CLEANUP_SQL),
1103        ] {
1104            let without_qualified_calls = sql.replace("pg_catalog.clock_timestamp()", "");
1105            assert!(
1106                !without_qualified_calls.contains("clock_timestamp()"),
1107                "{name} contains a search_path-resolved clock_timestamp() call"
1108            );
1109        }
1110    }
1111}