Skip to content

Commit bff1233

Browse files
author
Cary Huang
committed
updates
1 parent a35dd3d commit bff1233

File tree

7 files changed

+197
-96
lines changed

7 files changed

+197
-96
lines changed

‎docs/en/getting-started/quick_start.md

Lines changed: 32 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,7 @@ Successful creation will have a record under `synchdb_conninfo` table. More deta
4747

4848
## **Start a Connector**
4949

50-
Use `synchdb_start_engine_bgw()` function to start a connector worker. It takes one argument which is the connection name created above. This command will spawn a new background worker to connect to the heterogeneous database with the specified configurations.
51-
52-
For example, the following will spawn 2 background worker in PostgreSQL, one replicating from a MySQL database, the other from SQL Server
50+
Use `synchdb_start_engine_bgw()` function to start a connector worker. It takes one argument which is the connection name created above. This command will start the connector in the default snapshot mode called `initial`, which will replicate the schema of all designated tables and their initial data in PostgreSQL.
5351

5452
```sql
5553
select synchdb_start_engine_bgw('mysqlconn');
@@ -58,9 +56,9 @@ select synchdb_start_engine_bgw('oracleconn');
5856
```
5957

6058
## Check Connector Running State
61-
Use `synchdb_state_view()` to examine all connectors' running states.
59+
Use `synchdb_state_view()` to examine all connectors' running states.
6260

63-
See below for an example output:
61+
If everything works fine, we should see outputs similar to below.
6462
``` SQL
6563
postgres=# select * from synchdb_state_view;
6664
name | connector_type | pid | stage | state | err | last_dbz_offset
@@ -72,54 +70,41 @@ postgres=# select * from synchdb_state_view;
7270

7371
```
7472

75-
Column Details:
76-
77-
| fields | description |
78-
|-|-|
79-
| name | the associated connector info name created by `synchdb_add_conninfo()`|
80-
| connector_type | the type of connector (mysql, oracle, sqlserver...etc)|
81-
| pid | the PID of the connector worker process|
82-
| stage | the stage of the connector worker process|
83-
| state | the state of the connector. Possible states are: <br><br><ul><li>stopped - connector is not running</li><li>initializing - connector is initializing</li><li>paused - connector is paused</li><li>syncing - connector is regularly polling change events</li><li>parsing (the connector is parsing a received change event) </li><li>converting - connector is converting a change event to PostgreSQL representation</li><li>executing - connector is applying the converted change event to PostgreSQL</li><li>updating offset - connector is writing a new offset value to Debezium offset management</li><li>restarting - connector is restarting </li><li>dumping memory - connector is dumping JVM memory summary in log file </li><li>unknown</li></ul> |
84-
| err | the last error message encountered by the worker which would have caused it to exit. This error could originated from PostgreSQL while processing a change, or originated from Debezium running engine while accessing data from heterogeneous database. |
85-
| last_dbz_offset | the last Debezium offset captured by synchdb. Note that this may not reflect the current and real-time offset value of the connector engine. Rather, this is shown as a checkpoint that we could restart from this offeet point if needed.|
73+
If there is an error during connector start, perhaps due to connectivity, incorrect user credentials or source database settings, the error message shall be captured and displayed as `err` in the connector state output.
8674

87-
## **Check Connector Running Statistics**
75+
More on running states [here](https://docs.synchdb.com/monitoring/state_view/), and also running statistics [here](https://docs.synchdb.com/monitoring/stats_view/).
8876

89-
Use `synchdb_stats_view()` view to examine the statistic information of all connectors. These statistics record cumulative measurements about different types of change events a connector has processed so far. Currently these statistic values are stored in shared memory and not persisted to disk. Persist statistics data is a feature to be added in near future. The statistic information is updated at every successful completion of a batch and it contains several timestamps of the first and last change event within that batch. By looking at these timestamps, we can roughly tell the time it takes to finish processing the batch and the delay between when the data is generated, and processed by both Debezium and PostgreQSL.
77+
## Check the Tables and Data
78+
By default, the source tables will be replicated to a new schema under the current postgreSQL database with the same name as the source database name. We can update the search path to view the new tables in different schema.
9079

91-
See below for an example output:
80+
For example:
9281
```sql
93-
postgres=# select * from synchdb_stats_view;
94-
name | ddls | dmls | reads | creates | updates | deletes | bad_events | total_events | batches_done | avg_batch_size | first_src_ts | first_dbz_ts | first_pg_ts | last_src_ts | last_dbz_ts | last_pg_ts
95-
------------+------+------+-------+---------+---------+---------+------------+--------------+--------------+----------------+---------------+---------------+---------------+---------------+---------------+---------------
96-
oracleconn | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 2 | 1 | 2 | 1744398189000 | 1744398230893 | 1744398231243 | 1744398198000 | 1744398230950 | 1744398231244
97-
(1 row)
82+
postgres=# set search_path=public,free,inventory;
83+
SET
84+
postgres=# \d
85+
List of relations
86+
Schema | Name | Type | Owner
87+
-----------+--------------------+----------+--------
88+
free | orders | table | ubuntu
89+
inventory | addresses | table | ubuntu
90+
inventory | addresses_id_seq | sequence | ubuntu
91+
inventory | customers | table | ubuntu
92+
inventory | customers_id_seq | sequence | ubuntu
93+
inventory | geom | table | ubuntu
94+
inventory | geom_id_seq | sequence | ubuntu
95+
inventory | perf_test_1 | table | ubuntu
96+
inventory | products | table | ubuntu
97+
inventory | products_id_seq | sequence | ubuntu
98+
inventory | products_on_hand | table | ubuntu
99+
public | synchdb_att_view | view | ubuntu
100+
public | synchdb_attribute | table | ubuntu
101+
public | synchdb_conninfo | table | ubuntu
102+
public | synchdb_objmap | table | ubuntu
103+
public | synchdb_state_view | view | ubuntu
104+
public | synchdb_stats_view | view | ubuntu
105+
(17 rows)
98106
```
99107

100-
Column Details:
101-
102-
| fields | description |
103-
|-|-|
104-
| name | the associated connector info name created by `synchdb_add_conninfo()`|
105-
| ddls | number of DDLs operations completed |
106-
| dmls | number of DMLs operations completed |
107-
| reads | number of READ events completed during initial snapshot stage |
108-
| creates | number of CREATES events completed during CDC stage |
109-
| updates | number of UPDATES events completed during CDC stage |
110-
| deletes | number of DELETES events completed during CDC stage |
111-
| bad_events | number of bad events ignored (such as empty events, unsupported DDL events..etc) |
112-
| total_events | total number of events processed (including bad_events) |
113-
| batches_done | number of batches completed |
114-
| avg_batch_size | average batch size (total_events / batches_done) |
115-
| first_src_ts | the timestamp in nanoseconds when the last batch's first event is produced at the external database |
116-
| first_dbz_ts | the timestamp in nanoseconds when the last batch's first event is processed by Debezium Engine |
117-
| first_pg_ts | the timestamp in nanoseconds when the last batch's first event is applied to PostgreSQL |
118-
| last_src_ts | the timestamp in nanoseconds when the last batch's last event is produced at the external database |
119-
| last_dbz_ts | the timestamp in nanoseconds when the last batch's last event is processed by Debezium Engine |
120-
| last_pg_ts | the timestamp in nanoseconds when the last batch's last event is applied to PostgreSQL |
121-
122-
123108
## **Stop a Connector**
124109

125110
Use `synchdb_stop_engine_bgw()` SQL function to stop a running or paused connector worker. This function takes `conninfo_name` as its only parameter, which can be found from the output of `synchdb_get_state()` view.

‎docs/en/monitoring/state_view.md

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,28 @@
1-
# State View
1+
# Connector Running State
2+
3+
## Check Connector Running State
4+
Use `synchdb_state_view()` to examine all connectors' running states.
5+
6+
See below for an example output:
7+
``` SQL
8+
postgres=# select * from synchdb_state_view;
9+
name | connector_type | pid | stage | state | err | last_dbz_offset
10+
---------------+----------------+--------+---------------------+---------+----------+------------------------------------------------------------------------------------------------------
11+
sqlserverconn | sqlserver | 579820 | change data capture | polling | no error | {"commit_lsn":"0000006a:00006608:0003","snapshot":true,"snapshot_completed":false}
12+
mysqlconn | mysql | 579845 | change data capture | polling | no error | {"ts_sec":1741301103,"file":"mysql-bin.000009","pos":574318212,"row":1,"server_id":223344,"event":2}
13+
oracleconn | oracle | 580053 | change data capture | polling | no error | offset file not flushed yet
14+
(3 rows)
15+
16+
```
17+
18+
Column Details:
19+
20+
| fields | description |
21+
|-|-|
22+
| name | the associated connector info name created by `synchdb_add_conninfo()`|
23+
| connector_type | the type of connector (mysql, oracle, sqlserver...etc)|
24+
| pid | the PID of the connector worker process|
25+
| stage | the stage of the connector worker process|
26+
| state | the state of the connector. Possible states are: <br><br><ul><li>stopped - connector is not running</li><li>initializing - connector is initializing</li><li>paused - connector is paused</li><li>syncing - connector is regularly polling change events</li><li>parsing (the connector is parsing a received change event) </li><li>converting - connector is converting a change event to PostgreSQL representation</li><li>executing - connector is applying the converted change event to PostgreSQL</li><li>updating offset - connector is writing a new offset value to Debezium offset management</li><li>restarting - connector is restarting </li><li>dumping memory - connector is dumping JVM memory summary in log file </li><li>unknown</li></ul> |
27+
| err | the last error message encountered by the worker which would have caused it to exit. This error could originated from PostgreSQL while processing a change, or originated from Debezium running engine while accessing data from heterogeneous database. |
28+
| last_dbz_offset | the last Debezium offset captured by synchdb. Note that this may not reflect the current and real-time offset value of the connector engine. Rather, this is shown as a checkpoint that we could restart from this offeet point if needed.|

‎docs/en/monitoring/stats_view.md

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,36 @@
1-
# Statistics View
1+
# Connector Statistics
2+
3+
## **Check Connector Running Statistics**
4+
5+
Use `synchdb_stats_view()` view to examine the statistic information of all connectors. These statistics record cumulative measurements about different types of change events a connector has processed so far. Currently these statistic values are stored in shared memory and not persisted to disk. Persist statistics data is a feature to be added in near future. The statistic information is updated at every successful completion of a batch and it contains several timestamps of the first and last change event within that batch. By looking at these timestamps, we can roughly tell the time it takes to finish processing the batch and the delay between when the data is generated, and processed by both Debezium and PostgreQSL.
6+
7+
See below for an example output:
8+
```sql
9+
postgres=# select * from synchdb_stats_view;
10+
name | ddls | dmls | reads | creates | updates | deletes | bad_events | total_events | batches_done | avg_batch_size | first_src_ts | first_dbz_ts | first_pg_ts | last_src_ts | last_dbz_ts | last_pg_ts
11+
------------+------+------+-------+---------+---------+---------+------------+--------------+--------------+----------------+---------------+---------------+---------------+---------------+---------------+---------------
12+
oracleconn | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 2 | 1 | 2 | 1744398189000 | 1744398230893 | 1744398231243 | 1744398198000 | 1744398230950 | 1744398231244
13+
(1 row)
14+
```
15+
16+
Column Details:
17+
18+
| fields | description |
19+
|-|-|
20+
| name | the associated connector info name created by `synchdb_add_conninfo()`|
21+
| ddls | number of DDLs operations completed |
22+
| dmls | number of DMLs operations completed |
23+
| reads | number of READ events completed during initial snapshot stage |
24+
| creates | number of CREATES events completed during CDC stage |
25+
| updates | number of UPDATES events completed during CDC stage |
26+
| deletes | number of DELETES events completed during CDC stage |
27+
| bad_events | number of bad events ignored (such as empty events, unsupported DDL events..etc) |
28+
| total_events | total number of events processed (including bad_events) |
29+
| batches_done | number of batches completed |
30+
| avg_batch_size | average batch size (total_events / batches_done) |
31+
| first_src_ts | the timestamp in nanoseconds when the last batch's first event is produced at the external database |
32+
| first_dbz_ts | the timestamp in nanoseconds when the last batch's first event is processed by Debezium Engine |
33+
| first_pg_ts | the timestamp in nanoseconds when the last batch's first event is applied to PostgreSQL |
34+
| last_src_ts | the timestamp in nanoseconds when the last batch's last event is produced at the external database |
35+
| last_dbz_ts | the timestamp in nanoseconds when the last batch's last event is processed by Debezium Engine |
36+
| last_pg_ts | the timestamp in nanoseconds when the last batch's last event is applied to PostgreSQL |

0 commit comments

Comments
 (0)