Skip to content

Commit ffab9f6

Browse files
author
Cary Huang
committed
updates
1 parent b135268 commit ffab9f6

File tree

5 files changed

+297
-5
lines changed

5 files changed

+297
-5
lines changed

‎docs/en/getting-started/remote_database_setups.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,7 @@ GO
168168

169169
### **When Table Schema Changed While CDC is Enabled**
170170

171-
If a table has already been added to the CDC capture list and being captured by SynchDB already, any schema change that has happened to this table on SQLServer needs to be re-added back to the CDC capture list to generate a proper DDL ALTER TABLE event to SynchDB. Refer to [DDL Replication](https://docs.synchdb.com/user-guide/ddl_replication/) page for more information.
171+
If a table has already been added to the CDC capture list and being captured by SynchDB already, any schema change that has happened to this table on SQLServer needs to be re-added back to the CDC capture list to generate a proper DDL ALTER TABLE event to SynchDB. Refer to [DDL Replication](https://docs.synchdb.com/architecture/ddl_replication/) page for more information.
172172

173173
## **Set up Oracle for SynchDB**
174174

‎docs/en/tutorial/mysql_cdc_to_postgresql.md

Lines changed: 146 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,146 @@
1-
# MySQL CDC to PostgreSQL
1+
# MySQL CDC to PostgreSQL
2+
3+
## Prepare MySQL Databasea for SynchDB
4+
5+
Before SynchDB can be used to replicate from MySQL, MySQL needs to be configured according to the procedure outlined [here](https://docs.synchdb.com/getting-started/remote_database_setups/)
6+
7+
## Create a MySQL Connector
8+
9+
Create a connector that targets all the tables under `inventory` database in MySQL.
10+
```sql
11+
SELECT synchdb_add_conninfo(
12+
'mysqlconn', '127.0.0.1', 3306, 'mysqluser',
13+
'mysqlpwd', 'inventory', 'postgres',
14+
'null', 'null', 'mysql');
15+
```
16+
17+
## Initial Snapshot + CDC
18+
19+
Start the connector using `initial` mode will perform the initial snapshot of all designated tables (all in this case). After this is completed, the change data capture (CDC) process will begin to stream for new changes.
20+
21+
```sql
22+
SELECT synchdb_start_engine_bgw('mysqlconn', 'initial');
23+
24+
or
25+
26+
SELECT synchdb_start_engine_bgw('mysqlconn');
27+
```
28+
29+
The stage of this connector should be in `initial snapshot` the first time it runs:
30+
```sql
31+
postgres=# select * from synchdb_state_view;
32+
name | connector_type | pid | stage | state | err | last_dbz_offs
33+
et
34+
------------+----------------+--------+------------------+---------+----------+-----------------------------------
35+
-------------------------
36+
mysqlconn2 | mysql | 522195 | initial snapshot | polling | no error | {"ts_sec":1750375008,"file":"mysql
37+
-bin.000003","pos":1500}
38+
(1 row)
39+
40+
```
41+
42+
A new schema called `inventory` will be created and all tables streamed by the connector will be replicated under that schema.
43+
```sql
44+
postgres=# set search_path=public,inventory;
45+
SET
46+
postgres=# \d
47+
List of relations
48+
Schema | Name | Type | Owner
49+
-----------+-------------------------+----------+--------
50+
inventory | addresses | table | ubuntu
51+
inventory | addresses_id_seq | sequence | ubuntu
52+
inventory | customers | table | ubuntu
53+
inventory | customers_id_seq | sequence | ubuntu
54+
inventory | geom | table | ubuntu
55+
inventory | geom_id_seq | sequence | ubuntu
56+
inventory | orders | table | ubuntu
57+
inventory | orders_order_number_seq | sequence | ubuntu
58+
inventory | products | table | ubuntu
59+
inventory | products_id_seq | sequence | ubuntu
60+
inventory | products_on_hand | table | ubuntu
61+
public | synchdb_att_view | view | ubuntu
62+
public | synchdb_attribute | table | ubuntu
63+
public | synchdb_conninfo | table | ubuntu
64+
public | synchdb_objmap | table | ubuntu
65+
public | synchdb_state_view | view | ubuntu
66+
public | synchdb_stats_view | view | ubuntu
67+
(17 rows)
68+
69+
```
70+
71+
After the initial snapshot is completed, and at least one subsequent changes is received and processed, the connector stage shall change from `initial snapshot` to `Change Data Capture`.
72+
```sql
73+
postgres=# select * from synchdb_state_view;
74+
name | connector_type | pid | stage | state | err | last_dbz_o
75+
ffset
76+
------------+----------------+--------+---------------------+---------+----------+--------------------------------
77+
----------------------------
78+
mysqlconn2 | mysql | 522195 | change data capture | polling | no error | {"ts_sec":1750375008,"file":"my
79+
sql-bin.000003","pos":1500}
80+
(1 row)
81+
82+
```
83+
84+
This means that the connector is now streaming for new changes of the designated tables. Restarting the connector in `initial` mode will proceed replication since the last successful point and initial snapshot will not be re-run.
85+
86+
## Initial Snapshot Only and no CDC
87+
88+
Start the connector using `initial_only` mode will perform the initial snapshot of all designated tables (all in this case) only and will not perform CDC after.
89+
90+
```sql
91+
SELECT synchdb_start_engine_bgw('mysqlconn', 'initial_only');
92+
93+
```
94+
95+
The connector would still appear to be `polling` from the connector but no change will be captured because Debzium internally has stopped the CDC. You have the option to shut it down. Restarting the connector in `initial_only` mode will not rebuild the tables as they have already been built.
96+
97+
```sql
98+
postgres=# select * from synchdb_state_view;
99+
name | connector_type | pid | stage | state | err | last_dbz_offset
100+
------------+----------------+--------+------------------+---------+----------+-----------------------------
101+
mysqlconn2 | mysql | 522330 | initial snapshot | polling | no error | offset file not flushed yet
102+
(1 row)
103+
104+
```
105+
106+
## Capture Table Schema Only + CDC
107+
108+
Start the connector using `no_data` mode will perform the schema capture only, build the corresponding tables in PostgreSQL and it does not replicate existing table data (skip initial snapshot). After the schema capture is completed, the connector goes into CDC mode and will start capture subsequent changes to the tables.
109+
110+
```sql
111+
SELECT synchdb_start_engine_bgw('mysqlconn', 'no_data');
112+
113+
```
114+
115+
Restarting the connector in `no_data` mode will not rebuild the schema again, and it will resume CDC since the last successful point.
116+
117+
## CDC only
118+
119+
Start the connector using `never` will skip schema capture and initial snapshot entirely and will go to CDC mode to capture subsequent changes. Please note that the connector expects all the capture tables have been created in PostgreSQL prior to starting in `never` mode. If the tables do not exist, the connector will encounter an error when it tries to apply a CDC change to a non-existent table.
120+
121+
```sql
122+
SELECT synchdb_start_engine_bgw('mysqlconn', 'never');
123+
124+
```
125+
126+
Restarting the connector in `never` mode will resume CDC since the last successful point.
127+
128+
## Always do Initial Snapahot + CDC
129+
130+
Start the connector using `always` mode will always capture the schemas of capture tables, always redo the initial snapshot and then go to CDC. This is similar to a reset button because everything will be rebuilt using this mode. Use it with caution especially when you have large number of tables being captured, which could take a long time to finish. After the rebuild, CDC resumes as normal.
131+
132+
```sql
133+
SELECT synchdb_start_engine_bgw('mysqlconn', 'always');
134+
135+
```
136+
137+
However, it is possible to select partial tables to redo the initial snapshot by using the `snapshottable` option of the connector. Tables matching the criteria in `snapshottable` will redo the inital snapshot, if not, their initial snapshot will be skipped. If `snapshottable` is null or empty, by default, all the tables specified in `table` option of the connector will redo the initial snapshot under `always` mode.
138+
139+
This example makes the connector only redo the initial snapshot of `inventory.customers` table. All other tables will have their snapshot skipped.
140+
```sql
141+
UPDATE synchdb_conninfo
142+
SET data = jsonb_set(data, '{snapshottable}', '"inventory.customers"')
143+
WHERE name = 'mysqlconn';
144+
```
145+
146+
After the initial snapshot, CDC will begin. Restarting a connector in `always` mode will repeat the same process described above.

‎docs/zh/getting-started/remote_database_setups.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,7 @@ GO
168168

169169
### **当启用 CDC 时表架构发生更改**
170170

171-
如果某个表已添加到 CDC 捕获列表并已被 SynchDB 捕获，则需要将 SQLServer 上此表发生的任何架构更改重新添加回 CDC 捕获列表，以向 SynchDB 生成正确的 DDL ALTER TABLE 事件。有关更多信息，请参阅 [DDL 复制](https://docs.synchdb.com/zh/user-guide/ddl_replication/) 页面。
171+
如果某个表已添加到 CDC 捕获列表并已被 SynchDB 捕获，则需要将 SQLServer 上此表发生的任何架构更改重新添加回 CDC 捕获列表，以向 SynchDB 生成正确的 DDL ALTER TABLE 事件。有关更多信息，请参阅 [DDL 复制](https://docs.synchdb.com/zh/architecture/ddl_replication/) 页面。
172172

173173
## **为 SynchDB 设置 Oracle**
174174

‎docs/zh/tutorial/mysql_cdc_to_postgresql.md

Lines changed: 146 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,146 @@
1-
# MySQL CDC to PostgreSQL
1+
# MySQL CDC 到 PostgreSQL
2+
3+
## 为 SynchDB 准备 MySQL 数据库
4+
5+
在使用 SynchDB 从 MySQL 复制之前，需要按照[此处](https://docs.synchdb.com/getting-started/remote_database_setups/) 概述的步骤配置 MySQL
6+
7+
## 创建 MySQL 连接器
8+
9+
创建一个连接器，该连接器指向 MySQL 中 `inventory` 数据库下的所有表。
10+
```sql
11+
SELECT synchdb_add_conninfo(
12+
'mysqlconn', '127.0.0.1', 3306, 'mysqluser',
13+
'mysqlpwd', 'inventory', 'postgres',
14+
'null', 'null', 'mysql');
15+
```
16+
17+
## 初始快照 + CDC
18+
19+
使用 `initial` 模式启动连接器将对所有指定表（在本例中为所有表）执行初始快照。完成后，变更数据捕获 (CDC) 进程将开始流式传输新的变更。
20+
21+
```sql
22+
SELECT synchdb_start_engine_bgw('mysqlconn', 'initial');
23+
24+
或
25+
26+
SELECT synchdb_start_engine_bgw('mysqlconn');
27+
```
28+
29+
The stage of this connector should be in `initial snapshot` the first time it runs:
30+
```sql
31+
postgres=# select * from synchdb_state_view;
32+
name | connector_type | pid | stage | state | err | last_dbz_offs
33+
et
34+
------------+----------------+--------+------------------+---------+----------+-----------------------------------
35+
-------------------------
36+
mysqlconn2 | mysql | 522195 | initial snapshot | polling | no error | {"ts_sec":1750375008,"file":"mysql
37+
-bin.000003","pos":1500}
38+
(1 row)
39+
40+
```
41+
42+
A new schema called `inventory` will be created and all tables streamed by the connector will be replicated under that schema.
43+
```sql
44+
postgres=# set search_path=public,inventory;
45+
SET
46+
postgres=# \d
47+
List of relations
48+
Schema | Name | Type | Owner
49+
-----------+-------------------------+----------+--------
50+
inventory | addresses | table | ubuntu
51+
inventory | addresses_id_seq | sequence | ubuntu
52+
inventory | customers | table | ubuntu
53+
inventory | customers_id_seq | sequence | ubuntu
54+
inventory | geom | table | ubuntu
55+
inventory | geom_id_seq | sequence | ubuntu
56+
inventory | orders | table | ubuntu
57+
inventory | orders_order_number_seq | sequence | ubuntu
58+
inventory | products | table | ubuntu
59+
inventory | products_id_seq | sequence | ubuntu
60+
inventory | products_on_hand | table | ubuntu
61+
public | synchdb_att_view | view | ubuntu
62+
public | synchdb_attribute | table | ubuntu
63+
public | synchdb_conninfo | table | ubuntu
64+
public | synchdb_objmap | table | ubuntu
65+
public | synchdb_state_view | view | ubuntu
66+
public | synchdb_stats_view | view | ubuntu
67+
(17 rows)
68+
69+
```
70+
71+
初始快照完成后，如果至少有一条后续更改被接收并处理，连接器阶段将从“初始快照”切换为“变更数据捕获”。
72+
```sql
73+
postgres=# select * from synchdb_state_view;
74+
name | connector_type | pid | stage | state | err | last_dbz_o
75+
ffset
76+
------------+----------------+--------+---------------------+---------+----------+--------------------------------
77+
----------------------------
78+
mysqlconn2 | mysql | 522195 | change data capture | polling | no error | {"ts_sec":1750375008,"file":"my
79+
sql-bin.000003","pos":1500}
80+
(1 row)
81+
82+
```
83+
84+
这意味着连接器现在正在流式传输指定表的新更改。以“初始”模式重新启动连接器将从上次成功点开始继续复制，并且不会重新运行初始快照。
85+
86+
## 仅初始快照，无 CDC
87+
88+
使用 `initial_only` 模式启动连接器将仅对所有指定表（在本例中为全部）执行初始快照，之后将不再执行 CDC。
89+
90+
```sql
91+
SELECT synchdb_start_engine_bgw('mysqlconn', 'initial_only');
92+
93+
```
94+
95+
连接器似乎仍在“轮询”其他连接器，但不会捕获任何更改，因为 Debzium 内部已停止 CDC。您可以选择关闭它。在 `initial_only` 模式下重新启动连接器不会重建表，因为它们已经构建好了。
96+
97+
```sql
98+
postgres=# select * from synchdb_state_view;
99+
name | connector_type | pid | stage | state | err | last_dbz_offset
100+
------------+----------------+--------+------------------+---------+----------+-----------------------------
101+
mysqlconn2 | mysql | 522330 | initial snapshot | polling | no error | offset file not flushed yet
102+
(1 row)
103+
104+
```
105+
106+
## 仅捕获表模式 + CDC
107+
108+
使用 `no_data` 模式启动连接器将仅执行模式捕获，在 PostgreSQL 中构建相应的表，并且不会复制现有表数据（跳过初始快照）。模式捕获完成后，连接器将进入 CDC 模式，并开始捕获对表的后续更改。
109+
110+
```sql
111+
SELECT synchdb_start_engine_bgw('mysqlconn', 'no_data');
112+
113+
```
114+
115+
在 `no_data` 模式下重新启动连接器将不会再次重建模式，并且它将从上次成功点恢复 CDC。
116+
117+
## 仅 CDC
118+
119+
使用 `never` 模式启动连接器将完全跳过模式捕获和初始快照，并进入 CDC 模式以捕获后续更改。请注意，连接器要求在以 `never` 模式启动之前，所有捕获表都已在 PostgreSQL 中创建。如果表不存在，连接器在尝试将 CDC 更改应用于不存在的表时将遇到错误。
120+
121+
```sql
122+
SELECT synchdb_start_engine_bgw('mysqlconn', 'never');
123+
124+
```
125+
126+
以“never”模式重启连接器将从上次成功点开始恢复 CDC。
127+
128+
## 始终执行初始快照 + CDC
129+
130+
使用 `always` 模式启动连接器将始终捕获捕获表的模式，始终重做初始快照，然后转到 CDC。这类似于重置按钮，因为使用此模式将重建所有内容。请谨慎使用，尤其是在捕获大量表时，这可能需要很长时间才能完成。重建后，CDC 将恢复正常。
131+
132+
```sql
133+
SELECT synchdb_start_engine_bgw('mysqlconn', 'always');
134+
135+
```
136+
137+
但是，可以使用连接器的 `snapshottable` 选项选择部分表来重做初始快照。符合 `snapshottable` 中条件的表将重做初始快照，否则将跳过其初始快照。如果 `snapshottable` 为 null 或为空，默认情况下，连接器 `table` 选项中指定的所有表都将在 `always` 模式下重做初始快照。
138+
139+
此示例使连接器仅重做 `inventory.customers` 表的初始快照。所有其他表的快照将被跳过。
140+
```sql
141+
UPDATE synchdb_conninfo
142+
SET data = jsonb_set(data, '{snapshottable}', '"inventory.customers"')
143+
WHERE name = 'mysqlconn';
144+
```
145+
146+
初始快照完成后，CDC 将开始。在 `always` 模式下重新启动连接器将重复上述过程。

‎mkdocs.yml

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ plugins:
7575
State View: Vista de estado
7676
Start/Stop a Connector: Iniciar/Detener un conector
7777
Object Mapping Workflow: Flujo de trabajo de mapeo de objetos
78+
MySQL CDC to PostgreSQL: CDC de MySQL a PostgreSQL
7879
- locale: zh
7980
name: 中文
8081
site_name: SynchDB 文档
@@ -110,6 +111,7 @@ plugins:
110111
State View: 状态视图
111112
Start/Stop a Connector: 启动/停止连接器
112113
Object Mapping Workflow: 对象映射工作流程
114+
MySQL CDC to PostgreSQL: MySQL CDC 到 PostgreSQL
113115
- mike:
114116
version_selector: true
115117
css_dir: css
@@ -142,7 +144,7 @@ nav:
142144
- Attribute View: monitoring/attr_view.md
143145
- JVM Memory Usage: monitoring/jvm_mem.md
144146
- Tutorial:
145-
# - MySQL CDC to PostgreSQL: tutorial/mysql_cdc_to_postgresql.md
147+
- MySQL CDC to PostgreSQL: tutorial/mysql_cdc_to_postgresql.md
146148
# - SQL Server CDC to PostgreSQL: tutorial/sqlserver_cdc_to_postgresql.md
147149
# - Oracle CDC to PostgreSQL: tutorial/oracle_cdc_to_postgresql.md
148150
- Selective Table Replication: tutorial/selective_table_sync.md

0 commit comments

Comments
 (0)