clockpipe 0.4.1

Clickhouse Data Synchronization Pipeline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107

# clockpipe

![](https://img.shields.io/badge/language-Rust-red) ![](https://img.shields.io/badge/version-0.4.1-brightgreen) [![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/myyrakle/clockpipe/blob/master/LICENSE)

- An alternative to clickpipe for on-premise clickhouse users.
- Based on CDC, data from the original source is written to clickhouse.

## Supported Source

- PostgreSQL

## Install

Build from source code

```bash
git clone https://github.com/myyrakle/clockpipe
cd clockpipe
cargo install --path .
```

Using Cargo

```bash
cargo install clockpipe
```

Using Docker

```bash
sudo docker run -v $(pwd)/clockpipe-config.json:/app/config.json --network host myyrakle/clockpipe:v0.4.1
```

## PostgreSQL Setup

- Synchronization is implemented through PostgreSQL Publication.
- modify `postgresql.conf` and restart postgresql server.

```bash
postgres=# SHOW config_file;
                   config_file
-------------------------------------------------
 /opt/homebrew/var/postgresql@14/postgresql.conf
```

enable logical replica

```bash
sudo vim /opt/homebrew/var/postgresql@14/postgresql.conf
```

```
wal_level=logical

max_slot_wal_keep_size=-1
max_wal_size=10240
```

```bash
sudo systemctl restart postgresql
```

## How to Run

- Prepare config file ([example](./example.json))
- Enter the information about the PostgreSQL table you want to synchronize.

```json
    "tables": [
        {
            "schema_name": "public",
            "table_name": "foo"
        },
        {
            "schema_name": "public",
            "table_name": "nc_usr_account"
        }
    ]
```

- Then, Run it

```bash
clockpipe run --config-file ./clockpipe-config.json
```

- Pipe automatically creates and synchronizes tables in Clickhouse by querying table information.

- If you don't want the initial synchronization, use the skip_copy option. (CDC-based synchronization still works.)

```json
    "tables": [
        {
            "schema_name": "public",
            "table_name": "user_table",
            "skip_copy": true
        }
    ]
```

- You can also adjust the log level. You can set values such as error, warn, info, and debug to the "RUST_LOG" environment variable.

```
RUST_LOG=debug clockpipe run --config-file ./clockpipe-config.json
```

- Columns added from the source will also be automatically synchronized after the initial table link. (if restarted)