[tidb] add tidb cdc connector #898
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teckick
changed the title
leonardBang
reviewed
Mar 7, 2022
There was a problem hiding this comment.
Thanks @eastfisher for the contribution, I left some comments. Please let me know If you need any help.
...ava/com/ververica/cdc/connectors/tidb/table/RowDataTiKVChangeEventDeserializationSchema.java
Outdated
Show resolved
Hide resolved
flink-connector-tidb-cdc/src/main/java/org/tikv/common/codec/RowV2.java
Outdated
Show resolved
Hide resolved
flink-connector-tidb-cdc/src/main/java/org/tikv/common/codec/TableCodecV1.java
Outdated
Show resolved
Hide resolved
flink-connector-tidb-cdc/src/main/java/org/tikv/common/codec/TableCodecV2.java
Outdated
Show resolved
Hide resolved
flink-connector-tidb-cdc/src/main/java/org/tikv/common/codec/TableCodecV2.java
Outdated
Show resolved
Hide resolved
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| com.ververica.cdc.connectors.tidb.table.TiDBTableSourceFactory |
There was a problem hiding this comment.
please help add one more blank line
...ctor-tidb-cdc/src/test/java/com/ververica/cdc/connectors/tidb/table/TiDBConnectorITCase.java
Outdated
Show resolved
Hide resolved
Comment on lines
75
to
137
| public static final ConfigOption<String> TIKV_PD_ADDRESSES = | ||
| ConfigOptions.key(ConfigUtils.TIKV_PD_ADDRESSES) | ||
| .stringType() | ||
| .noDefaultValue() | ||
| .withDescription("TiKV cluster's PD address"); | ||
|
|
||
| public static final ConfigOption<Long> TIKV_GRPC_TIMEOUT = | ||
| ConfigOptions.key(ConfigUtils.TIKV_GRPC_TIMEOUT) | ||
| .longType() | ||
| .noDefaultValue() | ||
| .withDescription("TiKV GRPC timeout in ms"); | ||
|
|
||
| public static final ConfigOption<Long> TIKV_GRPC_SCAN_TIMEOUT = | ||
| ConfigOptions.key(ConfigUtils.TIKV_GRPC_SCAN_TIMEOUT) | ||
| .longType() | ||
| .noDefaultValue() | ||
| .withDescription("TiKV GRPC scan timeout in ms"); | ||
|
|
||
| public static final ConfigOption<Integer> TIKV_BATCH_GET_CONCURRENCY = | ||
| ConfigOptions.key(ConfigUtils.TIKV_BATCH_GET_CONCURRENCY) | ||
| .intType() | ||
| .noDefaultValue() | ||
| .withDescription("TiKV GRPC batch get concurrency"); | ||
|
|
||
| public static final ConfigOption<Integer> TIKV_BATCH_PUT_CONCURRENCY = | ||
| ConfigOptions.key(ConfigUtils.TIKV_BATCH_PUT_CONCURRENCY) | ||
| .intType() | ||
| .noDefaultValue() | ||
| .withDescription("TiKV GRPC batch put concurrency"); | ||
|
|
||
| public static final ConfigOption<Integer> TIKV_BATCH_SCAN_CONCURRENCY = | ||
| ConfigOptions.key(ConfigUtils.TIKV_BATCH_SCAN_CONCURRENCY) | ||
| .intType() | ||
| .noDefaultValue() | ||
| .withDescription("TiKV GRPC batch scan concurrency"); | ||
|
|
||
| public static final ConfigOption<Integer> TIKV_BATCH_DELETE_CONCURRENCY = | ||
| ConfigOptions.key(ConfigUtils.TIKV_BATCH_DELETE_CONCURRENCY) | ||
| .intType() | ||
| .noDefaultValue() | ||
| .withDescription("TiKV GRPC batch delete concurrency"); | ||
|
|
||
| public static TiConfiguration getTiConfiguration(final Map<String, String> options) { | ||
| final Configuration configuration = Configuration.fromMap(options); | ||
|
|
||
| final TiConfiguration tiConf = | ||
| configuration | ||
| .getOptional(TIKV_PD_ADDRESSES) | ||
| .map(TiConfiguration::createDefault) | ||
| .orElseGet(TiConfiguration::createDefault); | ||
|
|
||
| configuration.getOptional(TIKV_GRPC_TIMEOUT).ifPresent(tiConf::setTimeout); | ||
| configuration.getOptional(TIKV_GRPC_SCAN_TIMEOUT).ifPresent(tiConf::setScanTimeout); | ||
| configuration | ||
| .getOptional(TIKV_BATCH_GET_CONCURRENCY) | ||
| .ifPresent(tiConf::setBatchGetConcurrency); | ||
| configuration | ||
| .getOptional(TIKV_BATCH_PUT_CONCURRENCY) | ||
| .ifPresent(tiConf::setBatchPutConcurrency); | ||
| configuration | ||
| .getOptional(TIKV_BATCH_SCAN_CONCURRENCY) | ||
| .ifPresent(tiConf::setBatchScanConcurrency); | ||
| configuration | ||
| .getOptional(TIKV_BATCH_DELETE_CONCURRENCY) | ||
| .ifPresent(tiConf::setBatchDeleteConcurrency); | ||
| return tiConf; | ||
| } |
There was a problem hiding this comment.
We cannot list all valida TiKV parameters here, I think we should enable user to pass tikv.xx options just like Flink Kafka Connector enable user to properties.xx.
GOODBOY008
reviewed
Mar 9, 2022
...r-tidb-cdc/src/main/java/com/ververica/cdc/connectors/tidb/table/TiDBTableSourceFactory.java
Outdated
Show resolved
Hide resolved
GOODBOY008
reviewed
Mar 9, 2022
| FactoryUtil.createTableFactoryHelper(this, context); | ||
|
|
||
| final ReadableConfig config = helper.getOptions(); | ||
| String hostname = config.get(HOSTNAME); |
There was a problem hiding this comment.
Option TDBSourceOptions.TIKV_PD_ADDRESSES is required,but you do not add to TiDBTableSource‘s constructor parameter.
GOODBOY008
reviewed
Mar 14, 2022
.../java/com/ververica/cdc/connectors/tidb/table/RowDataTiKVEventDeserializationSchemaBase.java
Show resolved
Hide resolved
teckick
changed the title
|
cherry-pick and squash commits, then force push. |
|
Thanks @eastfisher and @GOODBOY008 for the great work! Looks pretty good to me as our first version, I'll help organize the commits and merge them. |
leonardBang
approved these changes
Mar 16, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes issue #243.