[TiDB-CDC] Support TiDB CDC connector

[peigenxiao]

mysql> select version();
+--------------------+
| version() |
+--------------------+
| 5.7.25-TiDB-v5.1.0 |
+--------------------+

like:
CREATE TEST_CDC (
ID STRING,
OPTY_ID STRING,
ACCEPT_DATE STRING,
ORG_ID STRING,
BRAND_LABEL STRING,
CURRENT_STATUS STRING,
PAID_AMOUNT STRING,
DEALER_ID STRING,
RECORD_STATUS STRING
)WITH (
'connector' = 'mysql-cdc',
'hostname' = 'xxxx',
'port' = '4000',
'username' = 'root',
'password' = 'xxxx',
'database-name' = 'flink',
'table-name' = 'test_cdc'
);

[Al-assad]

@peigenxiao
The CDC implementation mechanism is different for MySQL and TiDB. We have provided flink-cdc-tidb-connector support for TiDB in tidb-incubator/Tibigdata project which has not yet been officially released. It was actually implemented by way of HybridSource introduced in Flink-1.13.
follow in https://github.com/tidb-incubator/TiBigData/tree/master/flink

[zoucao]

hi @Al-assad， I take a cursory look at Tibigdata, it is a good project, combine batch and streaming, I found it use ticdc to collect the incremental data, right? Why don't use tikv-java-client and StreamObserver, such that we can get record from grpc directly just like the ticdc. What are your concerns or are there any disadvantages you don’t choose it ?

[Al-assad]

@zoucao The cdc-client of tikv-java-client was designed by @shanzi, used in TiFlink project and only recently merged into tikv-java-client project (tikv/client-java#174). Actually the unified streaming source design of TiBigData has been planned by @sunxiaoguang at an earlier time.
Both designs have their own advantages and disadvantages, you can access the discussion at here. welcome to discuss in more details in TiBigData community 👋🏻.

[zoucao]

Thanks for your replying, @Al-assad , serveral months ago, I made some changes based on tiflink and wanted it could be used in our internal, but I found it was not perfect and had many problems need to fix. From your perspective，which project will the community put more effort into ?

[Al-assad]

Thanks for your replying, @Al-assad , serveral months ago, I made some changes based on tiflink and wanted it could be used in our internal, but I found it was not perfect and had many problems need to fix. From your perspective，which project will the community put more effort into ?

personally speaking, both may co-exist to meet various scenarios. tikv-java makes application more streamlined while ticdc has greater isolation

[teckick]

@leonardBang Please assign to me

[leonardBang]

Thanks for picking up this ticket @eastfisher, assigned to you.