Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI Failure (key symptom) in EndToEndShadowIndexingTest.test_reset_from_cloud #17408

Closed
vbotbuildovich opened this issue Mar 26, 2024 · 3 comments
Assignees
Labels
area/storage auto-triaged used to know which issues have been opened from a CI job ci-failure sev/high loss of availability, pathological performance degradation, recoverable corruption

Comments

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Mar 26, 2024

https://buildkite.com/redpanda/redpanda/builds/46794

Module: rptest.tests.e2e_shadow_indexing_test
Class: EndToEndShadowIndexingTest
Method: test_reset_from_cloud
Arguments: {
    "cloud_storage_type": 1
}
test_id:    EndToEndShadowIndexingTest.test_reset_from_cloud
status:     FAIL
run time:   163.769 seconds

<BadLogLines nodes=docker-rp-15(1) example="ERROR 2024-03-26 16:29:40,647 [shard 1:au  ] cloud_storage - partition_manifest.cc:1036 - ["90000000/meta/kafka/panda-topic/0_24/manifest.bin"] New segment does not line up with previous segment: {anomaly_types: {{offset_gap}}, new_segment: {o=15897-16410 t={timestamp: 1711470563708}-{timestamp: 1711470563817}}, previous_segment: {{is_compacted: false, size_bytes: 1091437, base_offset: 13339, committed_offset: 13848, base_timestamp: {timestamp: 1711470562580}, max_timestamp: {timestamp: 1711470562714}, delta_offset: 12, ntp_revision: 24, archiver_term: 1, segment_term: 1, delta_offset_end: 12, sname_format: {v3}, metadata_size_hint: 0}}}">
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 269, in run_test
    return self.test_context.function(self.test)
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/mark/_mark.py", line 481, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/root/tests/rptest/services/cluster.py", line 173, in wrapped
    redpanda.raise_on_bad_logs(
  File "/root/tests/rptest/services/redpanda.py", line 1569, in raise_on_bad_logs
    raise BadLogLines(bad_lines)
rptest.services.utils.BadLogLines: <BadLogLines nodes=docker-rp-15(1) example="ERROR 2024-03-26 16:29:40,647 [shard 1:au  ] cloud_storage - partition_manifest.cc:1036 - ["90000000/meta/kafka/panda-topic/0_24/manifest.bin"] New segment does not line up with previous segment: {anomaly_types: {{offset_gap}}, new_segment: {o=15897-16410 t={timestamp: 1711470563708}-{timestamp: 1711470563817}}, previous_segment: {{is_compacted: false, size_bytes: 1091437, base_offset: 13339, committed_offset: 13848, base_timestamp: {timestamp: 1711470562580}, max_timestamp: {timestamp: 1711470562714}, delta_offset: 12, ntp_revision: 24, archiver_term: 1, segment_term: 1, delta_offset_end: 12, sname_format: {v3}, metadata_size_hint: 0}}}">

JIRA Link: CORE-1937

@vbotbuildovich vbotbuildovich added auto-triaged used to know which issues have been opened from a CI job ci-failure labels Mar 26, 2024
@dotnwat dotnwat added area/storage sev/high loss of availability, pathological performance degradation, recoverable corruption labels Apr 4, 2024
@nvartolomei
Copy link
Contributor

nvartolomei commented Jul 9, 2024

Logs are long gone and can't reproduce. Waiting for another occurrence.

@WillemKauf
Copy link
Contributor

Possibly fixed by #21348.

@WillemKauf
Copy link
Contributor

Closing due to potentially fixed issue and lack of logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage auto-triaged used to know which issues have been opened from a CI job ci-failure sev/high loss of availability, pathological performance degradation, recoverable corruption
Projects
None yet
Development

No branches or pull requests

4 participants