-
Notifications
You must be signed in to change notification settings - Fork 580
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cloud_storage: Fix transactions e2e test #6001
Conversation
Log frequent message on trace instead of debug.
Extend prefix logger header with segment base/max offsets.
Currently, tx-manifests are downloaded every time. Because of that we will make a request to S3 when the remote_segment is materialized. Even if transactions are not used we will still have to make a request to get NoSuchKey response. To avoid this this commit adds logic that seralizes the manifest to SI cache.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm but last commit could use an expanded commit message.
@@ -216,6 +219,7 @@ def done(): | |||
producer.produce(self.topic) | |||
except ck.KafkaException as err: | |||
self.logger.warn(f"producer error: {err}") | |||
time.sleep(10) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use short timeout if kafka producer returned an error. Add exception for
BadLogLines.
This commit message is describing the literal introduction of the sleep, not why the sleep is being added. Can you add the why to the commit message?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks!
Use short timeout if kafka producer returned an error. This is needed to guarantee that the previous transaction was aborted. The added delay is longer than the transaction's timeout. Add exception for BadLogLines.
1eee35a
to
7cdc775
Compare
\backport v22.1.x |
Cover letter
Fix test_restore_with_aborted_tx test failure on CI.
Use
transactions.timeout.ms
in kafka producer and add delay before reconnect.Fix segment hydration logic
remote_segment
. Currently, the tx-manifest is never saved to SI cache. This PR fixes this so the materialization of transactional metadata could be done from cache instead of S3.Backport Required
UX changes
none
Release notes
Improvements