Lightning-fast PostgreSQL to Meilisearch sync engine
Real-time data synchronization with automatic retries, parallel processing, and zero downtime
Features • Quick Start • Configuration • Contributing • Documentation
Core Capabilities
- 🚄 Real-time CDC - Sub-second data synchronization using PostgreSQL logical replication
- ⚡ High Performance - Process 10,000+ events/second with parallel work-stealing architecture
- 🔄 Automatic Recovery - Built-in retry mechanisms with exponential backoff and circuit breakers
- 💾 Persistent State - Redis-based checkpointing for seamless restarts and recovery
- 📊 Production Ready - Comprehensive metrics, health checks, and monitoring integrations
- 🎯 Flexible Mapping - Transform, filter, and enrich data with powerful pipeline configuration
- 🔌 Extensible - Plugin system for custom transformations and data processing
Data Integrity & Reliability
- ✅ Exactly-Once Delivery - Transaction-based checkpointing with event deduplication
- 🔐 Atomic Operations - Two-phase commit protocol ensures data consistency
- 🗄️ Multi-Source Support - Sync from multiple PostgreSQL databases simultaneously
- 🗑️ Soft Delete Handling - Configurable detection and transformation of soft deletes
- 📦 Dead Letter Queue - Automatic handling of failed events with retry policies
- 🔍 Snapshot Isolation - Consistent reads during full table synchronization
Performance Optimization
- 📈 Adaptive Batching - Dynamic batch sizing based on workload and latency
- 🧠 Smart Work Stealing - Automatic load balancing across parallel workers
- 💪 Connection Pooling - Optimized connection management for high throughput
- 🚦 Memory Efficient - Streaming processing with bounded memory usage
- ⏱️ Sub-100ms P50 Latency - Optimized for real-time synchronization
Operations & Monitoring
- 📡 Prometheus Metrics - Comprehensive metrics for monitoring and alerting
- 🔧 REST API - Full management API for runtime control and diagnostics
- 🏥 Health Checks - Liveness and readiness probes for container orchestration
- 📋 Event Replay - Replay events from specific checkpoints for recovery
- 🔍 Diagnostic Tools - Built-in debugging and troubleshooting endpoints
- 📚 Structured Logging - JSON-formatted logs with correlation IDs
🚀 Quick Start
Get MeiliBridge running in under 2 minutes!
Prerequisites
- PostgreSQL 10+ with logical replication enabled
- Meilisearch 1.0+ instance
- Docker (recommended) or Rust 1.70+ (for manual build)
PostgreSQL Setup
Before starting MeiliBridge, prepare your PostgreSQL database:
-- 1. Enable logical replication in postgresql.conf
-- wal_level = logical
-- max_replication_slots = 4
-- max_wal_senders = 4
-- 2. Create a user with replication privileges
;
-- 3. Grant necessary permissions on your database
CONNECT ON DATABASE your_database TO meilibridge;
USAGE ON SCHEMA public TO meilibridge;
SELECT ON ALL TABLES IN SCHEMA public TO meilibridge;
ALTER DEFAULT PRIVILEGES IN SCHEMA public SELECT ON TABLES TO meilibridge;
-- 4. Create publication for the tables you want to sync
CREATE PUBLICATION meilibridge_pub FOR TABLE users, products, orders;
-- Or for all tables:
-- CREATE PUBLICATION meilibridge_pub FOR ALL TABLES;
Note: MeiliBridge will automatically create the replication slot if configured with create_slot: true.
Docker (Recommended)
# Pull and run with minimal config
# Check health status
# View logs
Docker Compose (Full Stack)
# Clone the repository
# Copy example environment file
# Start PostgreSQL, Meilisearch, Redis, and MeiliBridge
# Verify all services are running
# Check synchronization status
Configuration File Setup
Create a config.yaml file:
# Minimal configuration
source:
type: postgresql
postgresql:
connection:
host: localhost
port: 5432
database: myapp
username: postgres
password: ${POSTGRES_PASSWORD}
meilisearch:
url: http://localhost:7700
api_key: ${MEILI_MASTER_KEY}
sync_tasks:
- table: users
index: users
primary_key: id
full_sync_on_start: true
Run with configuration:
# Using Docker
# Using binary
Manual Installation
# Clone repository
# Build in release mode
# Run with configuration
# Linux
# macOS (Intel)
# macOS (Apple Silicon)
# Make executable
# Run
Verify Installation
# Check version
# Validate configuration
# Generate sample configuration
# Start with debug logging
Next Steps
- 📚 Read the Getting Started Guide for detailed setup
- 📊 Set up monitoring with Prometheus
- 🚀 Deploy to production with our deployment guide
⚙️ Configuration
Create a config.yaml file with your settings:
# Basic connection settings
source:
type: postgresql
postgresql:
connection:
host: localhost
port: 5432
database: myapp
username: postgres
password: ${POSTGRES_PASSWORD} # Environment variable support
meilisearch:
url: http://localhost:7700
api_key: ${MEILI_MASTER_KEY}
# Define sync tasks
sync_tasks:
- table: users
index: users
primary_key: id
full_sync_on_start: true
Configuration Reference
PostgreSQL Source
source:
type: postgresql
postgresql:
connection:
host: localhost # PostgreSQL host
port: 5432 # PostgreSQL port
database: myapp # Database name
username: postgres # Username (needs REPLICATION privilege)
password: secret # Password (supports ${ENV_VAR})
# Replication settings
slot_name: meilibridge_slot # Replication slot name
publication_name: meilibridge_pub # Publication name
create_slot: true # Auto-create replication slot
# Connection pool settings
pool:
max_size: 10 # Maximum connections
min_idle: 2 # Minimum idle connections
acquire_timeout: 30 # Connection acquire timeout (seconds)
idle_timeout: 600 # Idle connection timeout (seconds)
max_lifetime: 1800 # Maximum connection lifetime (seconds)
# Statement cache
statement_cache:
enabled: true # Enable prepared statement caching
max_size: 100 # Maximum cached statements
Meilisearch Destination
meilisearch:
url: http://localhost:7700 # Meilisearch URL
api_key: masterKey # API key (supports ${ENV_VAR})
timeout: 30 # Request timeout (seconds)
# Retry settings
max_retries: 3 # Maximum retry attempts
retry_on_timeout: true # Retry on timeout errors
auto_create_index: true # Auto-create missing indexes
primary_key: id # Default primary key field
# Circuit breaker (fault tolerance)
circuit_breaker:
enabled: true # Enable circuit breaker
failure_threshold: 5 # Failures before opening
reset_timeout: 60 # Reset timeout (seconds)
sync_tasks:
- id: users_sync # Unique task ID
table: public.users # Source table (schema.table)
index: users # Target Meilisearch index
primary_key: id # Primary key field
# Sync behavior
full_sync_on_start: true # Perform full sync on startup
auto_start: true # Auto-start this task
# Filtering
filter:
event_types: # Event types to process
conditions:
- field: deleted
op: not_equals
value: true # Skip soft-deleted records
# Field transformations
transform:
fields:
email:
type: lowercase # Convert email to lowercase
full_name:
type: compute
expression: "concat(first_name, ' ', last_name)"
# Field mapping
mapping:
fields:
user_id: id # Rename user_id to id
created_at: created_timestamp
unmapped_fields_strategy: include # include/exclude/prefix
# Processing options
options:
batch_size: 1000 # Events per batch
batch_timeout_ms: 1000 # Batch timeout (milliseconds)
retry:
max_retries: 3 # Max retry attempts
initial_delay: 1000 # Initial retry delay (ms)
max_delay: 60000 # Maximum retry delay (ms)
multiplier: 2.0 # Backoff multiplier
Redis Configuration (Optional)
redis:
url: redis://localhost:6379 # Redis URL
password: ${REDIS_PASSWORD} # Redis password
database: 0 # Redis database number
key_prefix: meilibridge # Key prefix for all keys
pool:
max_size: 10 # Maximum connections
min_idle: 2 # Minimum idle connections
connection_timeout: 5 # Connection timeout (seconds)
Performance Tuning
performance:
parallel_processing:
enabled: true # Enable parallel processing
workers_per_table: 4 # Worker threads per table
max_concurrent_events: 100 # Max concurrent events
work_stealing: true # Enable work stealing
work_steal_interval_ms: 50 # Work steal check interval
buffer_size: 10000 # Event buffer size
checkpoint_interval: 10 # Checkpoint save interval (seconds)
API Server Configuration
api:
enabled: true # Enable REST API
host: 0.0.0.0 # API host
port: 7701 # API port
cors:
enabled: true # Enable CORS
origins: # Allowed origins
auth:
enabled: false # Enable authentication
type: bearer # Auth type (bearer)
tokens:
- name: admin
token: ${API_TOKEN} # Admin token
role: admin
Monitoring & Logging
logging:
level: info # Log level (trace/debug/info/warn/error)
format: pretty # Log format (pretty/json)
metrics:
enabled: true # Enable Prometheus metrics
port: 9090 # Metrics port
path: /metrics # Metrics endpoint
features:
auto_recovery: true # Auto-recover from failures
health_checks: true # Enable health endpoints
distributed_mode: false # Enable distributed mode
Environment Variables
All configuration values support environment variable substitution:
password: ${POSTGRES_PASSWORD}
api_key: ${MEILI_MASTER_KEY:-default_value} # With default
Common environment variables:
MEILIBRIDGE_CONFIG- Config file pathMEILIBRIDGE_LOG_LEVEL- Log levelPOSTGRES_PASSWORD- PostgreSQL passwordMEILI_MASTER_KEY- Meilisearch API keyREDIS_PASSWORD- Redis password
🔧 Command Line Options
)
)
🤝 Contributing
We love contributions! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.
Quick Contribution Guide
-
Fork & Clone
-
Set Up Development Environment
# Install Rust | # Install dependencies # Run tests -
Make Your Changes
- Create a feature branch:
git checkout -b feature/amazing-feature - Make your changes
- Add tests if applicable
- Ensure all tests pass:
cargo test
- Create a feature branch:
-
Submit Pull Request
- Commit your changes:
git commit -m 'Add amazing feature' - Push to your fork:
git push origin feature/amazing-feature - Open a pull request
- Commit your changes:
Development Resources
- 📖 Architecture Overview - Understand the codebase
- 🔧 API Development Guide - Add new API endpoints
- 🔄 CDC Operations Guide - Work with CDC components
- 🗺️ Roadmap - See what's planned
Code of Conduct
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
📚 Documentation
- Getting Started Guide - Detailed installation and setup
- Configuration & Architecture - Deep dive into configuration
- API Reference - REST API documentation
- CDC Operations - CDC setup and troubleshooting
- Roadmap - Future plans and features
🆘 Getting Help
- 📋 GitHub Issues - Report bugs or request features
- 💬 Discussions - Ask questions and share ideas
- 📧 Email: support@meilibridge.dev
📄 License
This project is licensed under the MIT License - see the LICENSE file for details.