Skip to main content

Crate pg_bus

Crate pg_bus 

Source
Expand description

pg-bus: a message bus on PostgreSQL.

Messages are published inside the caller’s transaction, kept in a backlog table, and delivered to subscribers that resume from a cursor.

§Example

use pg_bus::{Bus, Config, Filter, Item};
use serde_json::json;

let bus = Bus::start(pool.clone(), Config::default()).await?;

// Publish inside the transaction that makes the change: delivered once
// it commits, never if it rolls back.
let mut tx = pool.begin().await?;
bus.publish(&mut tx, "/topic/35", &json!({ "post_id": 52 }), None).await?;
tx.commit().await?;

// Subscribe from a position (here, from now on) to some channels, with
// the tags this subscriber holds.
let filter = Filter {
    channels: vec!["/topic/35".into()],
    tags: vec!["user:3".into()],
};
let mut subscription = bus.subscribe(bus.now().await?, filter);
while let Ok(item) = subscription.next().await {
    match item {
        Item::Message(m) => println!("{} {} {}", m.position, m.channel, m.data),
        Item::Gap => println!("missed some: reload"),
    }
}

§Ordering

Ids come from a sequence at insert, but transactions commit in their own order, so a cursor over ids alone would skip a message whose transaction commits after a later one was delivered. Each message therefore records the id of the transaction that wrote it (xid8), and a message is only delivered once every older transaction has finished: below the current snapshot’s xmin. Delivery runs in (xid, id) order, and since nothing can commit below the horizon afterwards, a Position in that order never skips a committed message. The cost is that a long-running write transaction anywhere in the database holds delivery back until it ends.

§Limits

  • PostgreSQL 13 or newer. Positions use xid8 and pg_current_xact_id; Bus::start refuses older servers.
  • One listening connection per process, held open for LISTEN. It must reach PostgreSQL directly or through a pooler in session mode: LISTEN does not work through pgbouncer in transaction mode. When the pool goes through one, set Config::listen_url to a direct URL for the listener.
  • Publish throughput is bounded by NOTIFY. A transaction that sends a notification takes a global lock at commit, so publishing transactions commit one at a time. BENCH.md has about 4,000 single-message transactions a second on a laptop; batching several messages into one transaction sends one notification.
  • No two-phase commit. Publishing queues a NOTIFY, and PostgreSQL refuses to PREPARE a transaction that has one.
  • Long write transactions delay delivery. A message is delivered once every older write transaction in the database has ended, so one left open (idle in transaction, a forgotten prepared transaction) holds back everything after it. Nothing is lost, and Bus::health and the listener’s warning name the transaction.
  • Gaps are not per channel. A subscriber resuming from before a trim is told it may have missed messages even when the trim was on another channel. It reloads when it need not, never the other way round.
  • Memory: each process keeps its most recent messages (Config::recent, default 10,000) for catch-ups, and each subscription queues at most a batch.
  • Channels match exactly; there are no wildcards or prefixes.
  • One order across channels: delivery follows transaction ids (roughly when each transaction first wrote), the same for every subscriber; a transaction that commits late is held back, not delivered out of turn. There are no per-channel sequence numbers.

Modules§

sseaxum
Delivery over HTTP with axum (the axum feature): server-sent events, and a long-poll fallback for networks that mangle streamed responses.

Structs§

Blocker
An open transaction holding delivery back.
Bus
The bus: publishing, the backlog, and one listener per process feeding its subscribers.
Config
How the bus runs.
Filter
Which messages a subscriber receives: those on its channels whose audience is everyone or shares a tag with the subscriber’s.
Health
What waits behind the delivery horizon, and why.
Message
A delivered message.
Position
Where a subscriber is in the bus: the last message it has seen, as (transaction id, message id). Opaque to clients, which hand back its string form (an SSE Last-Event-ID).
Stats
A snapshot of Bus::stats.
Subscription
Trim
How the background task trims: every every, deleting messages older than max_age and all but the newest keep_per_channel on each channel. MessageBus’s defaults: 7 days, 1000 per channel.

Enums§

Error
Item
What a subscription yields.

Functions§

stall_report
The warning the listener logs when the oldest waiting message has waited longer than threshold; None while delivery keeps up.