Skip to content

Commit 7a2e9a3

Browse files
author
Cary Huang
committed
updates
1 parent 870ccba commit 7a2e9a3

File tree

5 files changed

+372
-224
lines changed

5 files changed

+372
-224
lines changed

‎docs/en/tutorial/object_mapping_workflow.md

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# Object Mapping Workflow
2+
3+
## **Create a Connector and Start it in `schemasync` Mode**
4+
5+
`schemasync` is a special mode that makes the connector connects to remote database and attempt to sync only the schema of designated tables. After this is done, the connector is put to `paused` state and user is able to review all the tables and data types created using the default rules and make change if needed.
6+
7+
```sql
8+
SELECT synchdb_add_conninfo(
9+
'mysqlconn',
10+
'127.0.0.1',
11+
3306,
12+
'mysqluser',
13+
'mysqlpwd',
14+
'inventory',
15+
'postgres',
16+
'',
17+
'mysql');
18+
19+
SELECT synchdb_start_engine_bgw('mysqlconn', 'schemasync');
20+
```
21+
## **Ensure the connector is put to paused state**
22+
```sql
23+
SELECT name, connector_type, pid, stage, state FROM synchdb_state_view;
24+
name | connector_type | pid | stage | state
25+
---------------+----------------+--------+-------------+---------
26+
mysqlconn | mysql | 579845 | schema sync | polling
27+
28+
```
29+
30+
## **Review the tables created by default mapping rules**
31+
32+
```sql
33+
postgres=# select * from synchdb_att_view;
34+
name | type | attnum | ext_tbname | pg_tbname | ext_attname | pg_attname | ext_atttypename | pg_atttypename | transform
35+
-----------+-------+--------+----------------------------+----------------------------+-------------+-------------+-----------------+----------------+-----------
36+
mysqlconn | mysql | 1 | inventory.addresses | inventory.addresses | id | id | INT | int4 |
37+
mysqlconn | mysql | 2 | inventory.addresses | inventory.addresses | customer_id | customer_id | INT | int4 |
38+
mysqlconn | mysql | 3 | inventory.addresses | inventory.addresses | street | street | VARCHAR | varchar |
39+
mysqlconn | mysql | 4 | inventory.addresses | inventory.addresses | city | city | VARCHAR | varchar |
40+
mysqlconn | mysql | 5 | inventory.addresses | inventory.addresses | state | state | VARCHAR | varchar |
41+
mysqlconn | mysql | 6 | inventory.addresses | inventory.addresses | zip | zip | VARCHAR | varchar |
42+
mysqlconn | mysql | 7 | inventory.addresses | inventory.addresses | type | type | ENUM | text |
43+
mysqlconn | mysql | 1 | inventory.customers | inventory.customers | id | id | INT | int4 |
44+
mysqlconn | mysql | 2 | inventory.customers | inventory.customers | first_name | first_name | VARCHAR | varchar |
45+
mysqlconn | mysql | 3 | inventory.customers | inventory.customers | last_name | last_name | VARCHAR | varchar |
46+
mysqlconn | mysql | 4 | inventory.customers | inventory.customers | email | email | VARCHAR | varchar |
47+
mysqlconn | mysql | 1 | inventory.geom | inventory.geom | id | id | INT | int4 |
48+
mysqlconn | mysql | 2 | inventory.geom | inventory.geom | g | g | GEOMETRY | text |
49+
mysqlconn | mysql | 3 | inventory.geom | inventory.geom | h | h | GEOMETRY | text |
50+
mysqlconn | mysql | 1 | inventory.products | inventory.products | id | id | INT | int4 |
51+
mysqlconn | mysql | 2 | inventory.products | inventory.products | name | name | VARCHAR | varchar |
52+
mysqlconn | mysql | 3 | inventory.products | inventory.products | description | description | VARCHAR | varchar |
53+
mysqlconn | mysql | 4 | inventory.products | inventory.products | weight | weight | FLOAT | float4 |
54+
mysqlconn | mysql | 1 | inventory.products_on_hand | inventory.products_on_hand | product_id | product_id | INT | int4 |
55+
mysqlconn | mysql | 2 | inventory.products_on_hand | inventory.products_on_hand | quantity | quantity | INT | int4 |
56+
(20 rows)
57+
58+
```
59+
60+
## **Define custom mapping rules**
61+
User can use `synchdb_add_objmap` function to create custom mapping rules. It can be used to map table name, column name, data types and defines a data transform expression rule
62+
63+
```sql
64+
SELECT synchdb_add_objmap('mysqlconn','table','inventory.products','stuff');
65+
SELECT synchdb_add_objmap('mysqlconn','table','inventory.customers','schema1.people');
66+
SELECT synchdb_add_objmap('mysqlconn','column','inventory.customers.last_name','family_name');
67+
SELECT synchdb_add_objmap('mysqlconn','column','inventory.customers.email','contact');
68+
SELECT synchdb_add_objmap('mysqlconn','datatype','inventory.geom.g','geometry|0');
69+
SELECT synchdb_add_objmap('mysqlconn','datatype','inventory.orders.quantity','bigint|0');
70+
SELECT synchdb_add_objmap('mysqlconn','transform','inventory.products.name','''>>>>>'' || ''%d'' || ''<<<<<''');
71+
```
72+
73+
## **Review all object mapping rules created so far**
74+
```sql
75+
postgres=# select * from synchdb_objmap;
76+
name | objtype | enabled | srcobj | dstobj
77+
-----------+-----------+---------+-------------------------------+----------------------------
78+
mysqlconn | table | t | inventory.products | stuff
79+
mysqlconn | column | t | inventory.customers.last_name | family_name
80+
mysqlconn | column | t | inventory.customers.email | contact
81+
mysqlconn | table | t | inventory.customers | schema1.people
82+
mysqlconn | transform | t | inventory.products.name | '>>>>>' || '%d' || '<<<<<'
83+
mysqlconn | datatype | t | inventory.geom.g | geometry|0
84+
mysqlconn | datatype | t | inventory.orders.quantity | bigint|0
85+
(7 rows)
86+
87+
```
88+
89+
## **Reload the object mapping rules**
90+
Once all custom rules have been defined, we need to signal the connector to load them. This will cause the connector to read and apply the object mapping rules. If it sees a discrepancy between current PostgreSQL values and the object mapping values, it will attempt to correct the mapping.
91+
```sql
92+
SELECT synchdb_reload_objmap('mysqlconn');
93+
94+
```
95+
96+
## **Review `synchdb_att_view` again for changes**
97+
```sql
98+
SELECT * from synchdb_att_view;
99+
name | type | attnum | ext_tbname | pg_tbname | ext_attname | pg_attname | ext_atttypename | pg_atttypename | transform
100+
-----------+-------+--------+----------------------------+----------------------------+-------------+-------------+-----------------+----------------+----------------------------
101+
mysqlconn | mysql | 1 | inventory.addresses | inventory.addresses | id | id | INT | int4 |
102+
mysqlconn | mysql | 2 | inventory.addresses | inventory.addresses | customer_id | customer_id | INT | int4 |
103+
mysqlconn | mysql | 3 | inventory.addresses | inventory.addresses | street | street | VARCHAR | varchar |
104+
mysqlconn | mysql | 4 | inventory.addresses | inventory.addresses | city | city | VARCHAR | varchar |
105+
mysqlconn | mysql | 5 | inventory.addresses | inventory.addresses | state | state | VARCHAR | varchar |
106+
mysqlconn | mysql | 6 | inventory.addresses | inventory.addresses | zip | zip | VARCHAR | varchar |
107+
mysqlconn | mysql | 7 | inventory.addresses | inventory.addresses | type | type | ENUM | text |
108+
mysqlconn | mysql | 1 | inventory.customers | schema1.people | id | id | INT | int4 |
109+
mysqlconn | mysql | 2 | inventory.customers | schema1.people | first_name | first_name | VARCHAR | varchar |
110+
mysqlconn | mysql | 3 | inventory.customers | schema1.people | last_name | family_name | VARCHAR | varchar |
111+
mysqlconn | mysql | 4 | inventory.customers | schema1.people | email | contact | VARCHAR | varchar |
112+
mysqlconn | mysql | 1 | inventory.geom | inventory.geom | id | id | INT | int4 |
113+
mysqlconn | mysql | 2 | inventory.geom | inventory.geom | g | g | GEOMETRY | geometry |
114+
mysqlconn | mysql | 3 | inventory.geom | inventory.geom | h | h | GEOMETRY | text |
115+
mysqlconn | mysql | 1 | inventory.products | public.stuff | id | id | INT | int4 |
116+
mysqlconn | mysql | 2 | inventory.products | public.stuff | name | name | VARCHAR | varchar | '>>>>>' || '%d' || '<<<<<'
117+
mysqlconn | mysql | 3 | inventory.products | public.stuff | description | description | VARCHAR | varchar |
118+
mysqlconn | mysql | 4 | inventory.products | public.stuff | weight | weight | FLOAT | float4 |
119+
mysqlconn | mysql | 1 | inventory.products_on_hand | inventory.products_on_hand | product_id | product_id | INT | int4 |
120+
mysqlconn | mysql | 2 | inventory.products_on_hand | inventory.products_on_hand | quantity | quantity | INT | int8 |
121+
```
122+
123+
## **Resume the connector or redo the entire snapshot**
124+
Once the object mappings have been confirmed correct, we can resume the connector. Please note that, resume will proceed to streaming only the new table changes. The existing data of the tables will not be copied.
125+
```sql
126+
SELECT synchdb_resume_engine('mysqlconn');
127+
```
128+
129+
To capture the table's existing data, we can also redo the entire snapshot with the new object mapping rules:
130+
```sql
131+
SELECT synchdb_restart_connector('mysqlconn', 'always');
132+
```

0 commit comments

Comments
 (0)