meilibridge 0.1.6

High-performance PostgreSQL to Meilisearch connector
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
<div align="center">
<h1>MeiliBridge</h1>
</div>

<div align="center">

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=flat&logo=docker&logoColor=white)](https://hub.docker.com/r/binarytouch/meilibridge)
[![Rust](https://img.shields.io/badge/rust-%23000000.svg?style=flat&logo=rust&logoColor=white)](https://www.rust-lang.org/)
[![codecov](https://codecov.io/github/binary-touch/meilibridge/graph/badge.svg?token=E8W87QQO3G)](https://codecov.io/github/binary-touch/meilibridge)
[![GitHub Release](https://img.shields.io/github/v/release/binary-touch/meilibridge?style=flat&logo=github)](https://github.com/binary-touch/meilibridge/releases/latest)

**Lightning-fast PostgreSQL to Meilisearch sync engine**

Real-time data synchronization with automatic retries, parallel processing, and zero downtime

[Features](#-features) • [Quick Start](#-quick-start) • [Monitoring](#-monitoring--observability) • [Configuration](#-configuration) • [Contributing](#-contributing) • [Documentation](docs/)

</div>

---

### Core Capabilities

- **🚄 Real-time CDC** - Sub-second data synchronization using PostgreSQL logical replication
- **⚡ High Performance** - Process 10,000+ events/second with parallel work-stealing architecture
- **🔄 Automatic Recovery** - Built-in retry mechanisms with exponential backoff and circuit breakers
- **💾 Persistent State** - Redis-based checkpointing for seamless restarts and recovery
- **📊 Production Ready** - Comprehensive metrics, health checks, and monitoring integrations
- **🎯 Flexible Mapping** - Transform, filter, and enrich data with powerful pipeline configuration
- **🔌 Extensible** - Plugin system for custom transformations and data processing

### Data Integrity & Reliability

- **✅ At-Least-Once Delivery** - Transaction-based checkpointing with event deduplication to minimize duplicates
- **🔐 Atomic Operations** - Two-phase commit protocol ensures data consistency
- **🗄️ Multi-Source Support** - Sync from multiple PostgreSQL databases simultaneously
- **🗑️ Soft Delete Handling** - Configurable detection and transformation of soft deletes
- **📦 Dead Letter Queue** - Automatic handling of failed events with retry policies
- **🔍 Snapshot Isolation** - Consistent reads during full table synchronization
- **♻️ Checkpoint Management** - Automatic cleanup with retention policies and memory pressure triggers
- **🔄 CDC Resume** - Seamless resume from saved checkpoints after restart
- **⚡ Field Validation** - Automatic validation against PostgreSQL schema at startup

### Performance Optimization

- **📈 Adaptive Batching** - Dynamic batch sizing based on workload and latency
- **🧠 Smart Work Stealing** - Automatic load balancing across parallel workers
- **💪 Connection Pooling** - Optimized connection management for high throughput
- **🚦 Memory Efficient** - Streaming processing with bounded memory usage
- **⏱️ Sub-100ms P50 Latency** - Optimized for real-time synchronization
- **🎛️ Backpressure Control** - Automatic flow control with channel-based backpressure
- **📊 Memory Monitoring** - Real-time system memory tracking with adaptive batch sizing

### Operations & Monitoring

- **📡 Prometheus Metrics** - Comprehensive metrics for monitoring and alerting
- **🔧 REST API** - Full management API for runtime control and diagnostics
- **🏥 Health Checks** - Liveness and readiness probes for container orchestration
- **📋 Event Replay** - Replay events from specific checkpoints for recovery
- **🔍 Diagnostic Tools** - Built-in debugging and troubleshooting endpoints
- **📚 Structured Logging** - JSON-formatted logs with correlation IDs
- **🔄 Full Sync API** - Trigger full table synchronization via REST API
- **🔌 Startup Validation** - Comprehensive connectivity checks for all components

---

## 🚀 Quick Start

Get MeiliBridge running in under 2 minutes!

### Prerequisites

- PostgreSQL 10+ with logical replication enabled
- Meilisearch 1.0+ instance
- Docker (recommended) or Rust 1.70+ (for manual build)

### PostgreSQL Setup

Before starting MeiliBridge, prepare your PostgreSQL database:

```sql
-- 1. Enable logical replication in postgresql.conf
-- wal_level = logical
-- max_replication_slots = 4
-- max_wal_senders = 4

-- 2. Create a user with replication privileges
CREATE USER meilibridge WITH REPLICATION LOGIN PASSWORD 'your_password';

-- 3. Grant necessary permissions on your database
GRANT CONNECT ON DATABASE your_database TO meilibridge;
GRANT USAGE ON SCHEMA public TO meilibridge;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO meilibridge;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO meilibridge;

-- 4. Create publication for the tables you want to sync
CREATE PUBLICATION meilibridge_pub FOR TABLE users, products, orders;
-- Or for all tables:
-- CREATE PUBLICATION meilibridge_pub FOR ALL TABLES;
```

**Note**: MeiliBridge will automatically create the replication slot if configured with `create_slot: true`.

### Docker (Recommended)

```bash
# Pull and run with minimal config
docker run -d \
  --name meilibridge \
  -e POSTGRES_URL="postgresql://user:pass@host:5432/db" \
  -e MEILISEARCH_URL="http://localhost:7700" \
  -e MEILISEARCH_API_KEY="your-api-key" \
  -p 7701:7701 \
  binarytouch/meilibridge:latest

# Check health status
curl http://localhost:7701/health

# View logs
docker logs -f meilibridge
```

### Docker Compose (Full Stack)

```bash
# Clone the repository
git clone https://github.com/binary-touch/meilibridge.git
cd meilibridge

# Copy example environment file
cp .env.example .env

# Start PostgreSQL, Meilisearch, Redis, and MeiliBridge
docker-compose up -d

# Verify all services are running
docker-compose ps

# Check synchronization status
curl http://localhost:7701/api/v1/status
```

More reference at [Docker Documentation](docker/README.md)

### Configuration File Setup

Create a `config.yaml` file:

```yaml
# Minimal configuration
source:
  type: postgresql
  host: localhost
  port: 5432
  database: myapp
  username: postgres
  password: ${POSTGRES_PASSWORD}

meilisearch:
  url: http://localhost:7700
  api_key: ${MEILI_MASTER_KEY}

redis:
  url: redis://localhost:6379

sync_tasks:
  - table: users
    index: users
    primary_key: id
    full_sync_on_start: true
```

Run with configuration:

```bash
# Using Docker
docker run -d \
  --name meilibridge \
  -v $(pwd)/config.yaml:/config.yaml \
  -p 7701:7701 \
  binarytouch/meilibridge:latest --config /config.yaml

# Using binary
./meilibridge --config config.yaml
```

### Manual Installation

<details>
<summary><b>Build from Source</b></summary>

```bash
# Clone repository
git clone https://github.com/binary-touch/meilibridge.git
cd meilibridge

# Build in release mode
cargo build --release

# Run with configuration
./target/release/meilibridge --config config.yaml
```
</details>

<details>
<summary><b>Download Pre-built Binary</b></summary>

```bash
# Linux (x86_64)
curl -L https://github.com/binary-touch/meilibridge/releases/latest/download/meilibridge-linux-amd64.tar.gz -o meilibridge.tar.gz
tar -xzf meilibridge.tar.gz
chmod +x meilibridge

# macOS (Intel)
curl -L https://github.com/binary-touch/meilibridge/releases/latest/download/meilibridge-darwin-amd64.tar.gz -o meilibridge.tar.gz
tar -xzf meilibridge.tar.gz
chmod +x meilibridge

# macOS (Apple Silicon M1/M2/M3)
curl -L https://github.com/binary-touch/meilibridge/releases/latest/download/meilibridge-darwin-arm64.tar.gz -o meilibridge.tar.gz
tar -xzf meilibridge.tar.gz
chmod +x meilibridge

# Windows (PowerShell)
# Download the Windows binary
Invoke-WebRequest -Uri "https://github.com/binary-touch/meilibridge/releases/latest/download/meilibridge-windows-amd64.exe.zip" -OutFile "meilibridge.zip"
# Extract the zip file
Expand-Archive -Path "meilibridge.zip" -DestinationPath "."
# Run the executable
.\meilibridge.exe --config config.yaml

# Run on Unix-like systems
./meilibridge --config config.yaml
```

**Note**: 
- All Unix binaries are packaged as `.tar.gz` files
- Windows binary is packaged as `.zip` file
- Linux ARM64 can be built using `make build-linux-arm64` (see Building for ARM64 below)
</details>

### Verify Installation

```bash
# Check version
meilibridge --version

# Validate configuration
meilibridge validate --config config.yaml

# Generate sample configuration
meilibridge generate-sample > config.yaml

# Start with debug logging
meilibridge --config config.yaml --log-level debug
```

### Building for ARM64

<details>
<summary><b>Build Linux ARM64 Binary</b></summary>

```bash
# Install build dependencies (one-time setup)
make install-deps

# Build using Docker
make build-linux-arm64

# Package the binary
make package-linux-arm64

# The packaged binary will be in dist/meilibridge-linux-arm64.tar.gz
```

**Requirements:**
- Docker must be installed and running
- Docker buildx support (included in recent Docker versions)
</details>

<details>
<summary><b>Build Multi-Architecture Docker Images</b></summary>

```bash
# Build multi-arch Docker image locally (AMD64 + ARM64)
make docker-build-multiarch

# Build and push to Docker Hub
make docker-push

# Verify the multi-arch image
make docker-verify

# Custom registry and image name
DOCKER_USERNAME=myuser DOCKER_REGISTRY=ghcr.io make docker-push
```

The multi-architecture Docker image supports:
- `linux/amd64` - For standard x86_64 servers
- `linux/arm64` - For ARM64 servers (AWS Graviton, Apple Silicon, etc.)

Docker will automatically pull the correct architecture for your platform.
</details>

### Next Steps

- 📚 Read the [Getting Started Guide]docs/getting-started.md for detailed setup
- 📊 Set up [monitoring]#monitoring--observability with Prometheus
- 🚀 Check our [deployment examples]docker/ for production deployment

---

## 🎮 Quick Demo Start

```bash
# Clone and start the demo
git clone https://github.com/binary-touch/meilibridge.git
cd meilibridge/demo
./start.sh

# Search products (after ~30 seconds)
curl -X POST 'http://localhost:7700/indexes/products/search' \
  -H 'Authorization: Bearer masterKey123' \
  -H 'Content-Type: application/json' \
  -d '{"q":"laptop","offset":0,"limit":20}'
```

### Demo Features

- **Automatic Data Generation**: Simulates real-world e-commerce activity
- **Error Recovery**: Test fault tolerance by stopping/starting services  
- **Performance Testing**: Handle bulk operations and high-frequency updates
- **Visual Interface**: Browse and search data at http://localhost:24900

👉 **[Full Demo Documentation](demo/README.md)** - Detailed scenarios, troubleshooting, and advanced usage

---

## 📊 Monitoring & Observability

MeiliBridge provides comprehensive monitoring capabilities:

### Prometheus Metrics

```yaml
# Enable metrics in config.yaml
monitoring:
  metrics_enabled: true
  metrics_interval_seconds: 60
```

Available metrics:
- `meilibridge_events_processed_total` - Total events processed
- `meilibridge_events_failed_total` - Failed events count
- `meilibridge_sync_lag_seconds` - Replication lag in seconds
- `meilibridge_batch_size` - Current batch size
- `meilibridge_checkpoint_lag` - Checkpoint delay

### Health Endpoints

- `GET /health` - Overall system health
- `GET /health/liveness` - Kubernetes liveness probe
- `GET /health/readiness` - Kubernetes readiness probe
- `GET /health/:component` - Component-specific health (postgresql, meilisearch, redis)

### Grafana Dashboard

Import our [Grafana dashboard](monitoring/grafana-dashboard.json) for visualizing:
- Event throughput and latency
- Error rates and recovery metrics
- Resource utilization
- Sync task status

### REST API Endpoints

**Base URL**: `http://localhost:7701/api/v1`

#### Sync Task Management
- `GET /api/v1/tasks` - List all sync tasks
- `GET /api/v1/tasks/:id` - Get sync task details
- `POST /api/v1/tasks` - Create new sync task
- `PUT /api/v1/tasks/:id` - Update sync task
- `DELETE /api/v1/tasks/:id` - Delete sync task
- `POST /api/v1/tasks/:id/pause` - Pause sync task
- `POST /api/v1/tasks/:id/resume` - Resume sync task
- `POST /api/v1/tasks/:id/full-sync` - Trigger full table sync
- `GET /api/v1/tasks/:id/stats` - Get task statistics

#### CDC Control
- `POST /api/v1/cdc/pause` - Pause all CDC processing
- `POST /api/v1/cdc/resume` - Resume all CDC processing
- `GET /api/v1/cdc/status` - Get CDC status and replication info

#### Dead Letter Queue
- `GET /api/v1/dead-letters` - Get DLQ statistics
- `POST /api/v1/dead-letters/:task_id/reprocess` - Reprocess failed events

#### Cache Management
- `GET /api/v1/cache/stats` - Get statement cache statistics
- `POST /api/v1/cache/clear` - Clear statement cache

#### Parallel Processing
- `GET /api/v1/parallel/status` - Get parallel processing status
- `GET /api/v1/parallel/queues` - Get queue sizes for all tables

More info at [API Documentation](docs/api-development.md)

### Logging

Configure structured logging:

```yaml
logging:
  level: info  # trace, debug, info, warn, error
  format: json # json or pretty
```

Use correlation IDs to trace requests:
```bash
grep "correlation_id=abc123" logs.json
```

---

## ⚙️ Configuration

Create a `config.yaml` file with your settings:

```yaml
# Basic connection settings
source:
  type: postgresql
  host: localhost
  port: 5432
  database: myapp
  username: postgres
  password: ${POSTGRES_PASSWORD}  # Environment variable support

meilisearch:
  url: http://localhost:7700
  api_key: ${MEILI_MASTER_KEY}

redis:
  url: redis://localhost:6379

# Define sync tasks
sync_tasks:
  - table: users
    index: users
    primary_key: id
    full_sync_on_start: true
```

### Configuration Reference

<details>
<summary><b>Source Configuration</b></summary>

#### PostgreSQL Source

```yaml
source:
  type: postgresql
  # Connection parameters
  host: localhost               # PostgreSQL host
  port: 5432                    # PostgreSQL port
  database: myapp               # Database name
  username: postgres            # Username (needs REPLICATION privilege)
  password: ${POSTGRES_PASSWORD} # Password (supports ${ENV_VAR})
  
  # Replication settings
  slot_name: meilibridge_slot   # Replication slot name (default: "meilibridge")
  publication: meilibridge_pub  # Publication name (default: "meilibridge_pub")
  
  # Connection pool settings
  pool:
    max_size: 10                # Maximum connections
    min_idle: 1                 # Minimum idle connections
    connection_timeout: 30      # Connection timeout (seconds)
    idle_timeout: 600           # Idle connection timeout (seconds)
  
  # SSL/TLS configuration
  ssl:
    mode: disable               # disable, prefer, require, verify-ca, verify-full
    ca_cert: /path/to/ca.crt    # CA certificate path
    client_cert: /path/to/cert  # Client certificate
    client_key: /path/to/key    # Client key
  
  # Statement cache
  statement_cache:
    enabled: true               # Enable prepared statement caching
    max_size: 100               # Maximum cached statements
```

#### Multiple Sources (Multi-database)

```yaml
sources:
  - name: primary               # Unique source identifier
    type: postgresql
    host: primary.db.com
    port: 5432
    database: main
    username: replicator
    password: ${PRIMARY_PASSWORD}
    slot_name: meilibridge_primary
    publication: meilibridge_pub_primary
    
  - name: secondary
    type: postgresql
    host: secondary.db.com
    port: 5432
    database: analytics
    username: replicator
    password: ${SECONDARY_PASSWORD}
    slot_name: meilibridge_secondary
    publication: meilibridge_pub_secondary
```
</details>

<details>
<summary><b>Destination Configuration</b></summary>

#### Meilisearch Destination

```yaml
meilisearch:
  url: http://localhost:7700    # Meilisearch URL
  api_key: ${MEILI_MASTER_KEY} # API key (supports ${ENV_VAR})
  timeout: 30                   # Request timeout (seconds)
  max_connections: 10           # Connection pool size
  batch_size: 1000              # Batch size for bulk operations
  auto_create_index: true       # Auto-create missing indexes
  primary_key: id               # Default primary key field
  
  # Index settings template (applied to new indexes)
  index_settings:
    searchable_attributes: []   # Fields to search
    displayed_attributes: []    # Fields to return
    filterable_attributes: []   # Fields for filtering
    sortable_attributes: []     # Fields for sorting
    ranking_rules: []           # Custom ranking rules
    stop_words: []              # Stop words list
    synonyms: {}                # Synonyms mapping
  
  # Circuit breaker (fault tolerance)
  circuit_breaker:
    enabled: true               # Enable circuit breaker
    error_rate: 0.5             # Open circuit at 50% error rate
    min_request_count: 10       # Min requests before evaluation
    consecutive_failures: 5     # Or 5 consecutive failures
    timeout_secs: 60            # Time before half-open state
```
</details>

<details>
<summary><b>Sync Task Configuration</b></summary>

```yaml
sync_tasks:
  - id: users_sync              # Unique task ID
    source_name: primary        # Source name (for multi-source setups)
    table: public.users         # Source table (schema.table)
    index: users                # Target Meilisearch index
    primary_key: id             # Primary key field
    
    # Sync behavior
    full_sync_on_start: true    # Perform full sync on startup
    auto_start: true            # Auto-start this task
    
    # Soft delete detection
    soft_delete:
      field: status             # Field to check
      delete_values:            # Values indicating deletion
        - DELETED
        - INACTIVE
      handle_on_full_sync: true # Filter during full sync
      handle_on_cdc: true       # Convert to DELETE during CDC
    
    # Filtering
    filter:
      event_types: [create, update, delete]  # Event types to process
      conditions:
        - field: deleted
          op: not_equals
          value: true           # Skip soft-deleted records
    
    # Field transformations
    transform:
      fields:
        email:
          type: lowercase       # Convert email to lowercase
        full_name:
          type: compute
          expression: "concat(first_name, ' ', last_name)"
    
    # Field mapping
    mapping:
      fields:
        user_id: id            # Rename user_id to id
        created_at: created_timestamp
      unmapped_fields_strategy: include  # include/exclude/prefix
    
    # Processing options
    options:
      batch_size: 1000          # Events per batch
      batch_timeout_ms: 1000    # Batch timeout (milliseconds)
      retry:
        max_retries: 3          # Max retry attempts
        initial_delay: 1000     # Initial retry delay (ms)
        max_delay: 60000        # Maximum retry delay (ms)
        multiplier: 2.0         # Backoff multiplier
```
</details>

<details>
<summary><b>Advanced Configuration</b></summary>

#### Redis Configuration

```yaml
redis:
  url: redis://localhost:6379   # Redis URL
  password: ${REDIS_PASSWORD}   # Redis password
  database: 0                   # Redis database number
  key_prefix: meilibridge       # Key prefix for all keys
  
  pool:
    max_size: 10               # Maximum connections
    min_idle: 1                # Minimum idle connections
    connection_timeout: 5      # Connection timeout (seconds)
  
  # Checkpoint retention (New!)
  checkpoint_retention:
    max_checkpoints_per_task: 10  # Keep last 10 checkpoints per task
    cleanup_on_memory_pressure: true  # Auto-cleanup when memory is high
    memory_pressure_threshold: 80.0   # Trigger cleanup at 80% memory
```

#### Performance Tuning

```yaml
performance:
  parallel_processing:
    enabled: true              # Enable parallel processing
    workers_per_table: 4       # Worker threads per table
    max_concurrent_events: 1000 # Max concurrent events
    work_stealing: true        # Enable work stealing
    work_steal_interval_ms: 100 # Work steal check interval
    work_steal_threshold: 50   # Min queue size difference
  
  batch_processing:
    default_batch_size: 100    # Default batch size
    max_batch_size: 1000       # Maximum batch size
    min_batch_size: 10         # Minimum batch size
    batch_timeout_ms: 5000     # Batch timeout
    adaptive_batching: true    # Dynamic batch sizing
    
    adaptive_config:
      target_latency_ms: 1000  # Target processing time
      adjustment_factor: 0.2   # Adjustment aggressiveness (0-1)
      metric_window_size: 10   # Metrics to average
      adjustment_interval_ms: 5000 # Min time between adjustments
      memory_pressure_threshold: 80.0 # Memory % to reduce batch
      per_table_optimization: true # Per-table batch sizing
  
  connection_pool:
    max_connections: 20        # Max connections
    min_connections: 5         # Min connections
    connection_timeout: 30     # Timeout (seconds)
    idle_timeout: 600          # Idle timeout (seconds)
```

#### API Server Configuration

```yaml
api:
  enabled: true                # Enable REST API
  host: 0.0.0.0               # API host
  port: 7701                  # API port
  
  cors:
    enabled: true              # Enable CORS
    origins: ["*"]             # Allowed origins
  
  auth:
    enabled: false             # Enable authentication
    jwt_secret: ${JWT_SECRET}  # JWT secret key
    token_expiry: 3600         # Token expiry (seconds)
    api_keys:
      - name: admin
        key: ${API_KEY}        # API key
        permissions: ["read", "write", "admin"]
```

#### Monitoring & Logging

```yaml
logging:
  level: info                  # Log level (trace/debug/info/warn/error)
  format: pretty               # Log format (pretty/json)
  
monitoring:
  metrics_enabled: true        # Enable Prometheus metrics
  metrics_interval_seconds: 60 # Metrics collection interval
  health_checks_enabled: true  # Enable health checks
  health_check_interval_seconds: 30 # Health check interval

features:
  auto_recovery: true         # Auto-recover from failures
  health_checks: true         # Enable health endpoints
  metrics_export: true        # Export Prometheus metrics
  distributed_mode: false     # Enable distributed mode
```

#### At-Least-Once Delivery with Deduplication

```yaml
at_least_once_delivery:       # At-least-once delivery with deduplication
  enabled: true               # Enable at-least-once delivery
  deduplication_window: 10000 # Events to track for deduplication
  transaction_timeout_secs: 30 # Transaction timeout
  two_phase_commit: true      # Use two-phase commit protocol
  checkpoint_before_write: true # Atomic checkpoint before write
```

#### Error Handling

```yaml
error_handling:
  retry:
    enabled: true
    max_attempts: 3
    initial_backoff_ms: 100
    max_backoff_ms: 30000
    backoff_multiplier: 2.0
    jitter_factor: 0.1
  
  dead_letter_queue:
    enabled: true
    storage: memory           # memory or redis
    max_entries_per_task: 10000
    retention_hours: 24
    auto_reprocess_interval_minutes: 0 # 0 = disabled
  
  circuit_breaker:
    enabled: false            # Global circuit breaker
    failure_threshold_percent: 50
    min_requests: 10
    reset_timeout_seconds: 60
    half_open_max_requests: 3
```
</details>

### Environment Variables

All configuration values support environment variable substitution:

```yaml
password: ${POSTGRES_PASSWORD}
api_key: ${MEILI_MASTER_KEY:-default_value}  # With default
```

Common environment variables:
- `MEILIBRIDGE_CONFIG` - Config file path
- `MEILIBRIDGE_LOG_LEVEL` - Log level
- `POSTGRES_PASSWORD` - PostgreSQL password
- `MEILI_MASTER_KEY` - Meilisearch API key
- `REDIS_PASSWORD` - Redis password

---

## 🔧 Command Line Options

```bash
meilibridge [OPTIONS] [COMMAND]

OPTIONS:
    -c, --config <FILE>      Configuration file path
    -l, --log-level <LEVEL>  Log level (trace/debug/info/warn/error)
    -h, --help              Print help information
    -V, --version           Print version information

COMMANDS:
    run             Run the synchronization service (default)
    validate        Validate configuration file
    generate-sample Generate sample configuration
    version         Show version information
```

---

## 🤝 Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details on:

- 📝 How to submit bug reports and feature requests
- 🔧 Setting up your development environment
- 🚀 Our development workflow and coding standards
- ✅ Testing requirements and guidelines

For a quick start:
```bash
git clone https://github.com/YOUR_USERNAME/meilibridge.git
cd meilibridge
cargo build
cargo test
```

**Code of Conduct**: Please treat everyone with respect and kindness.

---

## 📚 Documentation

- **[Getting Started Guide]docs/getting-started.md** - Detailed installation and setup
- **[Configuration & Architecture]docs/configuration-architecture.md** - Deep dive into configuration
- **[API Reference]docs/api-development.md** - REST API documentation
- **[CDC Operations]docs/cdc-operations.md** - CDC setup and troubleshooting
- **[Contributing Guide]CONTRIBUTING.md** - How to contribute to the project

---

## 🆘 Getting Help

- 📋 [GitHub Issues]https://github.com/binary-touch/meilibridge/issues - Report bugs or request features
- 💬 [Discussions]https://github.com/binary-touch/meilibridge/discussions - Ask questions and share ideas
- 📖 [Documentation]docs/ - Browse all documentation

---

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

---

<div align="center">
Made with ❤️ in India
</div>