Expand description
pg-bus: a message bus on PostgreSQL.
Messages are published inside the caller’s transaction, kept in a backlog table, and delivered to subscribers that resume from a cursor.
§Example
use pg_bus::{Bus, Config, Filter, Item};
use serde_json::json;
let bus = Bus::start(pool.clone(), Config::default()).await?;
// Publish inside the transaction that makes the change: delivered once
// it commits, never if it rolls back.
let mut tx = pool.begin().await?;
bus.publish(&mut tx, "/topic/35", &json!({ "post_id": 52 }), None).await?;
tx.commit().await?;
// Subscribe from a position (here, from now on) to some channels, with
// the tags this subscriber holds.
let filter = Filter {
channels: vec!["/topic/35".into()],
tags: vec!["user:3".into()],
};
let mut subscription = bus.subscribe(bus.now().await?, filter);
while let Ok(item) = subscription.next().await {
match item {
Item::Message(m) => println!("{} {} {}", m.position, m.channel, m.data),
Item::Gap => println!("missed some: reload"),
}
}§Ordering
Ids come from a sequence at insert, but transactions commit in their own
order, so a cursor over ids alone would skip a message whose transaction
commits after a later one was delivered. Each message therefore records
the id of the transaction that wrote it (xid8), and a message is only
delivered once every older transaction has finished: below the current
snapshot’s xmin. Delivery runs in (xid, id) order, and since nothing
can commit below the horizon afterwards, a Position in that order
never skips a committed message. The cost is that a long-running write
transaction anywhere in the database holds delivery back until it ends.
§Limits
- PostgreSQL 13 or newer. Positions use
xid8andpg_current_xact_id;Bus::startrefuses older servers. - One listening connection per process, held open for LISTEN. It must
reach PostgreSQL directly or through a pooler in session mode: LISTEN
does not work through pgbouncer in transaction mode. When the pool goes
through one, set
Config::listen_urlto a direct URL for the listener. - Publish throughput is bounded by NOTIFY. A transaction that sends a notification takes a global lock at commit, so publishing transactions commit one at a time. BENCH.md has about 4,000 single-message transactions a second on a laptop; batching several messages into one transaction sends one notification.
- No two-phase commit. Publishing queues a NOTIFY, and PostgreSQL refuses to PREPARE a transaction that has one.
- Long write transactions delay delivery. A message is delivered once
every older write transaction in the database has ended, so one left
open (idle in transaction, a forgotten prepared transaction) holds back
everything after it. Nothing is lost, and
Bus::healthand the listener’s warning name the transaction. - Gaps are not per channel. A subscriber resuming from before a trim is told it may have missed messages even when the trim was on another channel. It reloads when it need not, never the other way round.
- Memory: each process keeps its most recent messages
(
Config::recent, default 10,000) for catch-ups, and each subscription queues at most a batch. - Channels match exactly; there are no wildcards or prefixes.
- One order across channels: delivery follows transaction ids (roughly when each transaction first wrote), the same for every subscriber; a transaction that commits late is held back, not delivered out of turn. There are no per-channel sequence numbers.
Modules§
- sse
axum - Delivery over HTTP with axum (the
axumfeature): server-sent events, and a long-poll fallback for networks that mangle streamed responses.
Structs§
- Blocker
- An open transaction holding delivery back.
- Bus
- The bus: publishing, the backlog, and one listener per process feeding its subscribers.
- Config
- How the bus runs.
- Filter
- Which messages a subscriber receives: those on its channels whose audience is everyone or shares a tag with the subscriber’s.
- Health
- What waits behind the delivery horizon, and why.
- Message
- A delivered message.
- Position
- Where a subscriber is in the bus: the last message it has seen, as
(transaction id, message id). Opaque to clients, which hand back its string form (an SSELast-Event-ID). - Stats
- A snapshot of
Bus::stats. - Subscription
- Trim
- How the background task trims: every
every, deleting messages older thanmax_ageand all but the newestkeep_per_channelon each channel. MessageBus’s defaults: 7 days, 1000 per channel.
Enums§
Functions§
- stall_
report - The warning the listener logs when the oldest waiting message has
waited longer than
threshold;Nonewhile delivery keeps up.