Module lsn_tracker

pg2any_lib

Module lsn_tracker 

Source
Expand description

Thread-safe LSN tracking for CDC replication

This module provides LSN tracking for CDC replication with file persistence:

  1. SharedLsnFeedback: Re-exported from pg_walstream - Thread-safe tracking of LSN positions for replication feedback to PostgreSQL

  2. LsnTracker: Thread-safe tracker for the last committed LSN with file persistence for graceful shutdown and restart recovery

§Simplified LSN Tracking

This implementation uses a simplified approach with a single flush_lsn value:

  • flush_lsn: Last WAL location successfully committed to destination database
    • Updated by consumer when transactions are successfully committed
    • Serves as the start_lsn for recovery on restart
    • Represents actual application of changes to the destination

§pg2any LSN Tracking Implementation

§File-Based Workflow

  • Consumer: Updates flush_lsn when transaction is successfully executed and committed
    • This happens in client.rs after successful execute_sql_batch() and delete_pending_transaction()
    • Represents that changes have been applied to the destination database
    • This LSN is persisted and used as start_lsn on restart

§Monitoring Replication Progress

You can monitor the replication progress using:

SELECT
    application_name,
    state,
    sent_lsn,
    write_lsn,
    flush_lsn,
    replay_lsn
FROM pg_stat_replication
WHERE application_name = 'pg2any';

§Optimization Strategy

Instead of persisting on every commit (which causes excessive I/O), the LsnTracker:

  • Batches multiple commits and persists periodically (default: every 1 second)
  • Uses a background tokio task for non-blocking persistence
  • Tracks a “dirty” flag to skip unnecessary writes
  • Ensures graceful shutdown with final persistence

Structs§

CdcMetadata
CDC replication metadata stored in the persistence file
ConsumerState
Consumer execution state for recovery
LsnTracker
Thread-safe tracker for the last committed LSN with file persistence
LsnTracking
LSN position for CDC replication tracking
SharedLsnFeedback
Thread-safe tracker for LSN positions used in replication feedback

Functions§

create_lsn_tracker_with_load_async
Create a shared LSN tracker with initial load from file