[CDCSDK] Migrate all the CDCSDK tests to use Publication/Replication Slot API #19599
Closed
1 task done
Assignees
Labels
area/cdcsdk
CDC SDK
kind/new-feature
This is a request for a completely new feature
priority/high
High Priority
Comments
yugabyte-ci
added
the
kind/enhancement
dr0pdb
added a commit
that referenced
this issue
Dec 5, 2023
…on Slots Summary: We need to update our unit tests to use the YSQL Replication Slots APIs. This is the first revision in that direction which targets the Consistent Stream tests. The tests have been updated with the following strategy: 1. Explicit checkpoint - Directly converted to use the Replication Slot API for creation 2. Implicit checkpoint - The Replication Slot API does not support implicit checkpoint as it is being deprecated. So for every such test, two tests have been added, one with explicit and another with implicit. The implicit one remains as it is while the explicit one uses the Replication Slot API Jira: DB-8377 Test Plan: Jenkins: test regex: .*CDCSDKConsistentStreamTest.* The tests that have been updated are: ``` ./yb_build.sh --cxx-test cdcsdk_consistent_stream-test --gtest_filter CDCSDKConsistentStreamTest.TestCDCSDKConsistentStreamWithTabletSplitExplicit ./yb_build.sh --cxx-test cdcsdk_consistent_stream-test --gtest_filter CDCSDKConsistentStreamTest.TestCDCSDKConsistentStreamWithTabletSplitImplicit ./yb_build.sh --cxx-test cdcsdk_consistent_stream-test --gtest_filter CDCSDKConsistentStreamTest.CDCSDKMultipleAlterExplicit ./yb_build.sh --cxx-test cdcsdk_consistent_stream-test --gtest_filter CDCSDKConsistentStreamTest.CDCSDKMultipleAlterImplicit ./yb_build.sh --cxx-test cdcsdk_consistent_stream-test --gtest_filter CDCSDKConsistentStreamTest.TestCDCSDKConsistentStreamWithRandomReqSafeTimeChanges ``` Reviewers: skumar, asrinivasan Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D30714
dr0pdb
added a commit
that referenced
this issue
Dec 7, 2023
Summary: We need to update our unit tests to use the YSQL Replication Slots APIs. This is the second revision in that direction which targets the Tablet Split tests. The tests have been updated with the following strategy: 1. Explicit checkpoint - Directly converted to use the Replication Slot API for creation 2. Implicit checkpoint - The Replication Slot API does not support implicit checkpoint as it is being deprecated. So for every such test, two tests have been added, one with explicit and another with implicit. The implicit one remains as it is while the explicit one uses the Replication Slot API Jira: DB-8377 Test Plan: Jenkins: test regex: .*CDCSDKTabletSplitTest.* Reviewers: asrinivasan, skumar Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D30724
dr0pdb
added a commit
that referenced
this issue
Dec 13, 2023
Summary: We need to update our unit tests to use the YSQL Replication Slots APIs. This is the third revision in that direction which targets some of the YSQL tests. There are many tests in the cdcsdk_ysql-test, so I'll do that in phases. The tests have been updated with the following strategy: 1. Explicit checkpoint - Directly converted to use the Replication Slot API for creation 2. Implicit checkpoint - The Replication Slot API does not support implicit checkpoint as it is being deprecated. So for every such test, two tests have been added, one with explicit and another with implicit. The implicit one remains as it is while the explicit one uses the Replication Slot API Jira: DB-8377 Test Plan: Jenkins: test regex: .*CDCSDKYsqlTest.* ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCDCLagMetricExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCDCLagMetricImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestGarbageCollectionFlagExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestGarbageCollectionFlagImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestGarbageCollectionWithSmallIntervalExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestGarbageCollectionWithSmallIntervalImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestGarbageCollectionWithLargerIntervalExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestGarbageCollectionWithLargerIntervalImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestNoGarbageCollectionBeforeIntervalExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestNoGarbageCollectionBeforeIntervalImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestExtendingIntentRetentionTimeExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestExtendingIntentRetentionTimeImplicit Reviewers: asrinivasan, skumar Reviewed By: asrinivasan Subscribers: ycdcxcluster Tags: #jenkins-ready Differential Revision: https://phorge.dev.yugabyte.com/D30966
yugabyte-ci
added
kind/new-feature
dr0pdb
added a commit
that referenced
this issue
Dec 26, 2023
Summary: We need to update our unit tests to use the YSQL Replication Slots APIs. This is the fourth revision in that direction which targets some of the YSQL tests. There are many tests in the cdcsdk_ysql-test, so the migration is being done in phases. The tests have been updated with the following strategy: Explicit checkpoint - Directly converted to use the Replication Slot API for creation Implicit checkpoint - The Replication Slot API does not support implicit checkpoint as it is being deprecated. For every such test, I've determined whether it depends on the CheckpointType. If yes, then I've added both variations for the test. Otherwise, I've updated it to just use Explicit checkpoint. The individual tests are listed in the test plan below. Jira: DB-8377 Test Plan: Jenkins: test regex: .*CDCSDKYsqlTest.* Tests that were written for both Explicit and Implicit checkpoints ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestSetCDCCheckpointExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestSetCDCCheckpointImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestMultipleStreamOnSameTabletExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestMultipleStreamOnSameTabletImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestMultipleActiveStreamOnSameTabletExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestMultipleActiveStreamOnSameTabletImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestActiveAndInactiveStreamOnSameTabletExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestActiveAndInactiveStreamOnSameTabletImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCheckpointPersistencyAllNodesRestartExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCheckpointPersistencyAllNodesRestartImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestIntentCountPersistencyAllNodesRestartExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestIntentCountPersistencyAllNodesRestartImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestHighIntentCountPersistencyAllNodesRestartExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestHighIntentCountPersistencyAllNodesRestartImplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestIntentCountPersistencyBootstrapExplicit ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestIntentCountPersistencyBootstrapImplicit Tests which were already with Explicit checkpoint. Just moved to use replication slot syntax. ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestLogGCForNewTablesAddedAfterCreateStream ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestGetCheckpointOnSnapshotBootstrapExplicit Tests that were moved from Implicit checkpoint to Explicit checkpoint ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupSingleStreamSingleTserver ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupSingleStreamMultiTserver ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupMultiStreamDeleteSingleStreamSingleTserver ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupMultiStreamDeleteSingleStreamMultiTserver ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupMultiStreamDeleteAllStreamsSingleTserver ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestCleanupMultiStreamDeleteAllStreamsMultiTserver ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestEnum ./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestEnumOnRestart Reviewers: skumar, asrinivasan Reviewed By: asrinivasan Subscribers: ycdcxcluster Differential Revision: https://phorge.dev.yugabyte.com/D31009
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-8377
Description
Migrate all the CDCSDK tests to use Publication/Replication Slot API
Source connector version
NA
Connector configuration
NA
YugabyteDB version
NA
Issue Type
kind/enhancement
Warning: Please confirm that this issue does not contain any sensitive information
The text was updated successfully, but these errors were encountered: