Segfault in `SchemaRegistryBasicAuthTest`.`test_delete_subject`, `BasicAuthScaleTest`.`test_many_users` #6903

Lazin · 2022-10-24T14:47:46Z

Version & Environment

Redpanda version: (use rpk version): dev

Link - https://buildkite.com/redpanda/redpanda/builds/17108#0184026e-77dd-4367-984f-286288a816d2/6-1398

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 44, in wrapped
    self.redpanda.raise_on_crash()
  File "/root/tests/rptest/services/redpanda.py", line 1313, in raise_on_crash
    raise NodeCrash(crashes)
rptest.services.utils.NodeCrash: <NodeCrash docker-rp-18: ==19932==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f3211951976 bp 0x7ffe01ab78e0 sp 0x7ffe01ab7640 T0)

The crash:

TRACE 2022-10-23 03:24:32,375 [shard 0] kafka - request_context.h:168 - [172.16.16.16:55818] sending 36:sasl_authenticate for {test_client}, response {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
DEBUG 2022-10-23 03:24:32,375 [shard 0] kafka/client - broker.h:65 - Dispatch: sasl_authenticate res: {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
DEBUG 2022-10-23 03:24:32,389 [shard 0] kafka/client - broker.h:65 - Dispatch: sasl_authenticate res: {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
DEBUG 2022-10-23 03:24:32,389 [shard 0] kafka/client - broker.h:65 - Dispatch: sasl_authenticate res: {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
AddressSanitizer:DEADLYSIGNAL
=================================================================
==19932==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f3211951976 bp 0x7ffe01ab78e0 sp 0x7ffe01ab7640 T0)
==19932==The signal is caused by a READ memory access.
==19932==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

The text was updated successfully, but these errors were encountered:

NyaliaLui · 2022-10-26T20:51:45Z

It looks like we de-referenced a pointer that was garbage or something like that. Unfortunately there is no other log messages or a backtrace to get more info so I'm going to submit a PR to enable debug logging for pandaproxy/schema-registry

BenPope · 2022-10-27T02:47:28Z

It looks like we de-referenced a pointer that was garbage or something like that. Unfortunately there is no other log messages or a backtrace to get more info so I'm going to submit a PR to enable debug logging for pandaproxy/schema-registry

I may have missed something: d5f6e9f

piyushredpanda · 2022-11-02T05:10:35Z

@NyaliaLui are you handling this? It is a segfault so needs some love.

Fix redpanda-data#6903 Signed-off-by: Ben Pope <ben@redpanda.com>

NyaliaLui · 2022-11-02T13:22:19Z

@piyushredpanda
@BenPope and I are tag-teaming this and it seems he found the solution. There is a fix in BenPope@b46f8fb

BenPope · 2022-11-02T14:39:32Z

@piyushredpanda
@BenPope and I are tag-teaming this and it seems he found the solution. There is a fix in BenPope@b46f8fb

It might be related, but it's not a fix yet. Well, 3 hours of running ~1500 tests on my machine was clear, but on CI maybe 10% failed.

dlex · 2022-11-09T03:45:05Z

another one in a feature branch:
https://buildkite.com/redpanda/redpanda/builds/18170#01845939-b983-410b-b03c-cbdd635e6ac7

BenPope · 2022-11-10T16:40:53Z

This seems to be most repeatable during the initial connect when using sasl. The crash is usually somewhere around:

redpanda/src/v/kafka/client/sasl_client.cc

Line 164 in a2517c2

client_final.set_proof(ScramAlgo::client_proof(

IME the best test for triggering it is:

redpanda/tests/rptest/tests/pandaproxy_test.py

Line 1461 in a2517c2

def test_many_users(self, num_users: int):

abhijat · 2022-11-14T07:45:02Z

On debug ci run for #7242 a test failed which seems to be similar to this issue - the test is different test_delete_subject_version ~~within the same test suite~~ within SchemaRegistryAutoAuthTest, but the error looks to be similar :

https://buildkite.com/redpanda/redpanda/builds/18462#01846631-734d-497c-b944-51a1cbf90aaf

test_id:    rptest.tests.schema_registry_test.SchemaRegistryAutoAuthTest.test_delete_subject_version
status:     FAIL
run time:   9.706 seconds
 
    <NodeCrash docker-rp-4: ==18503==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f530974b976 bp 0x7ffc60c1ab60 sp 0x7ffc60c1a8c0 T0)
>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
...
...

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 50, in wrapped
    self.redpanda.raise_on_crash()
  File "/root/tests/rptest/services/redpanda.py", line 1407, in raise_on_crash
    raise NodeCrash(crashes)
rptest.services.utils.NodeCrash: <NodeCrash docker-rp-4: ==18503==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f530974b976 bp 0x7ffc60c1ab60 sp 0x7ffc60c1a8c0 T0)
>

log from docker-rp-4 where the error occured:

DEBUG 2022-11-11 12:45:20,403 [shard 0] kafka/client - broker.h:75 - Dispatch from node 3: sasl_handshake res: {error_code={ error_code: none [0] } mechanisms={SCRAM-SHA-256, SCRAM-SHA-512}}
DEBUG 2022-11-11 12:45:20,403 [shard 0] kafka/client - broker.h:68 - Dispatch to node 1: sasl_authenticate req: {auth_bytes=****}
DEBUG 2022-11-11 12:45:20,405 [shard 0] kafka/client - broker.h:68 - Dispatch to node 2: sasl_authenticate req: {auth_bytes=****}
DEBUG 2022-11-11 12:45:20,406 [shard 0] kafka/client - broker.h:68 - Dispatch to node 3: sasl_authenticate req: {auth_bytes=****}
DEBUG 2022-11-11 12:45:20,409 [shard 0] security - scram_authenticator.cc:24 - Received client first message {client_first_message}
DEBUG 2022-11-11 12:45:20,411 [shard 0] kafka/client - broker.h:75 - Dispatch from node 1: sasl_authenticate res: {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
DEBUG 2022-11-11 12:45:20,415 [shard 0] security - authorizer.h:66 - Adding ACL binding: {pattern type {topic} name {_schemas} pattern {literal} entry {principal {type ephemeral user name __schema_registry} host {{any_host}} op all perm allow}}
DEBUG 2022-11-11 12:45:20,417 [shard 0] kafka/client - broker.h:75 - Dispatch from node 2: sasl_authenticate res: {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
Reactor stalled for 32 ms on shard 0. Backtrace: 0x357184da 0x48036134 0x4803560c 0x47e11923 0x47e0ba94 0x47e0b537 0x47e0bcf8 0x47e1176a 0x42abf 0x379cb341 0x379ce853 0x379cdf4a 0x3caabc48 0x35c4e370 0x35c4de95 0x47e31a6d 0x47e378ce 0x47e3c42f 0x47e39fb8 0x47b2999c 0x47b27055 0x357dfa94 0x3579ce6f 0x2d58f 0x2d648 0x356dc9a4
kernel callstack:
DEBUG 2022-11-11 12:45:20,449 [shard 0] kafka/client - broker.h:68 - Dispatch to node 2: sasl_authenticate req: {auth_bytes=****}
DEBUG 2022-11-11 12:45:20,451 [shard 0] kafka/client - broker.h:75 - Dispatch from node 3: sasl_authenticate res: {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
AddressSanitizer:DEADLYSIGNAL
=================================================================
==18503==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f530974b976 bp 0x7ffc60c1ab60 sp 0x7ffc60c1a8c0 T0)
==18503==The signal is caused by a READ memory access.
==18503==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

NyaliaLui · 2022-11-14T13:53:35Z

On debug ci run for #7242 a test failed which seems to be similar to this issue - the test is different test_delete_subject_version ~~within the same test suite~~ within SchemaRegistryAutoAuthTest, but the error looks to be similar :

Yeah that's the same error.

andijcr · 2022-11-14T15:30:30Z

I think this might be another instance of this issue
https://buildkite.com/redpanda/redpanda/builds/18531#01847486-2519-426a-bc36-06470615bd0c

graphcareful · 2022-11-16T18:31:57Z

Another instance here: https://buildkite.com/redpanda/redpanda/builds/18686#0184812d-4a62-411e-aa0a-51fa424f2105

dlex · 2022-11-21T21:35:33Z

Another instance in a feature branch, and very similar to what @abhijat has reported:
https://buildkite.com/redpanda/redpanda/builds/18876#01849b69-4d18-4866-a1f8-8246ccd5e26b

test_id:    rptest.tests.schema_registry_test.SchemaRegistryAutoAuthTest.test_serde_client
status:     FAIL
run time:   9.782 seconds


    <NodeCrash docker-rp-3: ==22772==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f27dc2ac976 bp 0x7fffadc24360 sp 0x7fffadc240c0 T0)
>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

.....
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 50, in wrapped
    self.redpanda.raise_on_crash()
  File "/root/tests/rptest/services/redpanda.py", line 1411, in raise_on_crash
    raise NodeCrash(crashes)
rptest.services.utils.NodeCrash: <NodeCrash docker-rp-3: ==22772==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f27dc2ac976 bp 0x7fffadc24360 sp 0x7fffadc240c0 T0)
>

docker-rp-3:

DEBUG 2022-11-21 20:47:09,553 [shard 0] kafka/client - sasl_client.cc:45 - Connecting to broker -1 with authentication: SCRAM-SHA-512:Y7b6zQfIrpL0eHfO5PyEiwCNxP3BL12q
DEBUG 2022-11-21 20:47:09,553 [shard 0] kafka/client - broker.h:68 - Dispatch to node -1: sasl_handshake req: {mechanism=SCRAM-SHA-512}
DEBUG 2022-11-21 20:47:09,555 [shard 0] kafka/client - broker.h:75 - Dispatch from node -1: sasl_handshake res: {error_code={ error_code: none [0] } mechanisms={SCRAM-SHA-256, SCRAM-SHA-512}}
DEBUG 2022-11-21 20:47:09,556 [shard 0] kafka/client - broker.h:68 - Dispatch to node -1: sasl_authenticate req: {auth_bytes=****}
DEBUG 2022-11-21 20:47:09,558 [shard 0] kafka/client - broker.h:75 - Dispatch from node -1: sasl_authenticate res: {error_code={ error_code: sasl_authentication_failed [58] } error_message={SASL authentication failed: security: Invalid credentials} auth_bytes=**** session_lifetime_ms=0}
WARN  2022-11-21 20:47:09,579 [shard 0] pandaproxy - service.cc:211 - mitigate_error: kafka::client::broker_error ({ node: -1 }, { error_code: sasl_authentication_failed [58] }: SASL authentication failed: security: Invalid credentials)
TRACE 2022-11-21 20:47:09,579 [shard 0] pandaproxy - service.cc:228 - inform: -1
Reactor stalled for 32 ms on shard 0. Backtrace: 0x3585915a 0x46d27474 0x46d2694c 0x46b02c63 0x46afcdd4 0x46afc877 0x46afd038 0x46b02aaa 0x42abf 0x358a03bf 0x37225b7d 0x37b285d0 0x37b2be64 0x37b2a492 0x4311dd4f 0x430e6182 0x43112e23 0x381a3970 0x381a35c5 0x46b22dad 0x46b28c0e 0x46b2d76f 0x46b2b2f8 0x4681acdc 0x46818395 0x35920714 0x358ddaef 0x2d58f 0x2d648 0x3581d624
kernel callstack:
AddressSanitizer:DEADLYSIGNAL
=================================================================
==22772==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f27dc2ac976 bp 0x7fffadc24360 sp 0x7fffadc240c0 T0)
==22772==The signal is caused by a READ memory access.
==22772==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

piyushredpanda · 2022-11-23T14:17:27Z

I assume @BenPope is chasing this? I'll remove other assignees so it's clear.

jcsp · 2022-11-23T19:08:17Z

FAIL test: BasicAuthScaleTest.test_many_users.num_users=500 (1/47 runs)
failure at 2022-11-22T15:29:01.731Z: <NodeCrash docker-rp-20: ==28922==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f93de43a976 bp 0x7f93d94bd7e0 sp 0x7f93d94bd540 T3)
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/18919#01849f4c-57cb-4516-b76b-37128ac44ad9

mmaslankaprv · 2022-11-30T11:42:22Z

FAIL test: RackAwarePlacementTest.test_replica_placement.rack_layout_str=xxYYzz.num_partitions=50.replication_factor=5.num_topics=2 (1/32 runs)
failure at 2022-11-29T11:37:13.713Z: <NodeCrash docker-rp-9: ==6482==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f5d3d6d6976 bp 0x7ffc5a78abe0 sp 0x7ffc5a78a940 T0)
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/19213#0184c296-3a23-4c65-b636-f67f792c0ed0

ballard26 · 2022-12-01T22:12:49Z

Another instance of the failure here
https://buildkite.com/redpanda/redpanda/builds/19344#0184cf20-383d-4fc5-9254-7ca07a8d7695

jcsp · 2022-12-02T09:10:43Z

This is a segfault in SchemaRegistryAutoAuthTest.test_mixed_deletes
https://buildkite.com/redpanda/redpanda/builds/19351#0184cf75-d509-42e0-8a45-44285fe849d4

@BenPope I'm tagging onto this issue because it seems like the general vibe is that there's some sort of SR crash buried that can affect various tests intermittently. This feels like a case where a new stress test may be needed to throw enough traffic at the system to reproduce it more reliably, if we have exhausted other avenues of investigation.

jcsp · 2022-12-12T14:00:55Z

Here's a segfault in test_post_subjects_subject_versions that I assume is the same underlying issue.

FAIL test: SchemaRegistryBasicAuthTest.test_post_subjects_subject_versions (1/55 runs)
failure at 2022-12-12T07:10:48.014Z: <NodeCrash docker-rp-11: ==7549==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f33b102f976 bp 0x7ffc17bcfc20 sp 0x7ffc17bcf980 T0)
on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/19625#01850509-0b3e-430c-bd35-26a8d74a613a

dotnwat · 2022-12-16T15:22:20Z

i'm working on this

dotnwat · 2022-12-16T23:04:41Z

yikes i dunno what's happening here

dotnwat · 2022-12-18T18:29:30Z

Effectively all logs containing instances of this crash happen near in time to sasl scram authentication. For instance, the following is typical. Here a panda proxy client is authenticating with a broker, which may or may not be the current broker.

Notice the reactor stall.

DEBUG 2022-11-11 12:45:20,417 [shard 0] kafka/client - broker.h:75 - Dispatch from node 2: sasl_authenticate res: {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
Reactor stalled for 32 ms on shard 0. Backtrace: 0x357184da 0x48036134 0x4803560c 0x47e11923 0x47e0ba94 0x47e0b537 0x47e0bcf8 0x47e1176a 0x42abf 0x379cb341 0x379ce853 0x379cdf4a 0x3caabc48 0x35c4e370 0x35c4de95 0x47e31a6d 0x47e378ce 0x47e3c42f 0x47e39fb8 0x47b2999c 0x47b27055 0x357dfa94 0x3579ce6f 0x2d58f 0x2d648 0x356dc9a4
kernel callstack:
DEBUG 2022-11-11 12:45:20,449 [shard 0] kafka/client - broker.h:68 - Dispatch to node 2: sasl_authenticate req: {auth_bytes=****}
DEBUG 2022-11-11 12:45:20,451 [shard 0] kafka/client - broker.h:75 - Dispatch from node 3: sasl_authenticate res: {error_code={ error_code: none [0] } error_message={nullopt} auth_bytes=**** session_lifetime_ms=0}
AddressSanitizer:DEADLYSIGNAL
=================================================================
==18503==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f530974b976 bp 0x7ffc60c1ab60 sp 0x7ffc60c1a8c0 T0)
==18503==The signal is caused by a READ memory access.
==18503==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer: nested bug in the same thread, aborting.

For a very long time we were not able to get any additional information other than this sanitizer output. Finally, by changing the ASAN_OPTIONS to disable the normal ASAN handling of the SEGV we were able to delegate directly normal fatal signal handling and get a core dump to analyze.

Below I've added handle_segv=0 which instructs ASAN to not register its SEGV handler.

diff --git a/tests/rptest/services/redpanda.py b/tests/rptest/services/redpanda.py
index 5440f700e..6b100f697 100644
--- a/tests/rptest/services/redpanda.py
+++ b/tests/rptest/services/redpanda.py
@@ -646,7 +646,7 @@ class RedpandaService(Service):
         # enable asan abort / core dumps by default
         self._environment = dict(
             ASAN_OPTIONS=
-            "abort_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1")
+            "verbosity=1:handle_segv=0:abort_on_error=1:disable_coredump=0")
         if environment is not None:
             self._environment.update(environment)

Below is the backtrace for all the seastar reactor threads. Finally we start to get some hints like this one in which it looks like some of the reactor stall handling and avx2 (related scram authentication primitives like sha256) are interacting?

Thread 1 (Thread 0x7f1769b2f6c0 (LWP 53712)):
#0  _Unwind_GetPtr (index=16, context=0x7f1769b18b80) at ../../../libgcc/unwind-dw2.c:251
#1  uw_update_context (fs=0x7f1769b18c70, context=0x7f1769b18b80) at ../../../libgcc/unwind-dw2.c:1546
#2  _Unwind_Backtrace (trace=0x7f176dc98d40 <backtrace_helper>, trace_argument=0x7f1769b18e30) at ../../../libgcc/unwind.inc:326
#3  0x00007f176dc98e43 in __GI___backtrace (array=<optimized out>, size=100) at backtrace.c:78
#4  0x0000562c09b2303b in __interceptor_backtrace () at /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4277
#5  0x0000562c1a908815 in seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}&&) (func=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
#6  0x0000562c1a907ced in seastar::backtrace_buffer::append_backtrace_oneline (this=0x7f1769b19da0) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:768
#7  0x0000562c1a6e3ff4 in seastar::print_with_backtrace (buf=..., oneline=true) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:787
#8  0x0000562c1a6de165 in seastar::internal::cpu_stall_detector::generate_trace (this=0x61100000ff40) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1346
#9  0x0000562c1a6ddc08 in seastar::internal::cpu_stall_detector::maybe_report (this=0x61100000ff40) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1088
#10 0x0000562c1a6de3c9 in seastar::internal::cpu_stall_detector::on_signal (this=0x61100000ff40) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1105
#11 0x0000562c1a6e3e3b in seastar::reactor::block_notifier () at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1329
#12 <signal handler called>
#13 sha256_block_data_order_avx2 () at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/accelerated/x86/elf/sha256-ssse3-x86_64.s:4597
Backtrace stopped: Cannot access memory at address 0x8095f88982a5b4e

Below is the full backtrace. Threads 2-4 are doing kafka::client stuff, and the the other threads seem to be asleep.

Thread 8 (Thread 0x7f1768ad56c0 (LWP 53714)):
#0  __GI___libc_read (nbytes=8, buf=0x7f1768ac5d00, fd=16) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=16, buf=0x7f1768ac5d00, nbytes=8) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x0000562c09b02a00 in __interceptor_read () at /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1024
#3  0x0000562c1ab47387 in seastar::thread_pool::work (this=0x61d000003c80, name=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/thread_pool.cc:41
#4  0x0000562c1ab4f167 in seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0::operator()() const (this=0x60300001f4e8) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/thread_pool.cc:29
#5  0x0000562c1ab4ef71 in std::__1::__invoke<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&>(seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&) (__f=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
#6  0x0000562c1ab4ee2d in std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&>(seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&) (__args=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:61
#7  0x0000562c1ab4ed95 in std::__1::__function::__alloc_func<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0, std::__1::allocator<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>, void ()>::operator()() (this=0x60300001f4e8) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:180
#8  0x0000562c1ab4a281 in std::__1::__function::__func<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0, std::__1::allocator<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>, void ()>::operator()() (this=0x60300001f4e0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:354
#9  0x0000562c09f752c6 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x60400000c590) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:507
#10 0x0000562c09f73fd9 in std::__1::function<void ()>::operator()() const (this=0x60400000c590) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:1184
#11 0x0000562c1a5aafe9 in seastar::posix_thread::start_routine (arg=0x60400000c590) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
#12 0x00007f176dc0b14d in start_thread (arg=<optimized out>) at pthread_create.c:442
#13 0x00007f176dc8ca00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x7f17682d46c0 (LWP 53717)):
#0  __GI___libc_read (nbytes=8, buf=0x7f17682c4d00, fd=17) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=17, buf=0x7f17682c4d00, nbytes=8) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x0000562c09b02a00 in __interceptor_read () at /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1024
#3  0x0000562c1ab47387 in seastar::thread_pool::work (this=0x61d00001fe80, name=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/thread_pool.cc:41
#4  0x0000562c1ab4f167 in seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0::operator()() const (this=0x603000020448) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/thread_pool.cc:29
#5  0x0000562c1ab4ef71 in std::__1::__invoke<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&>(seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&) (__f=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
#6  0x0000562c1ab4ee2d in std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&>(seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&) (__args=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:61
#7  0x0000562c1ab4ed95 in std::__1::__function::__alloc_func<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0, std::__1::allocator<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>, void ()>::operator()() (this=0x603000020448) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:180
#8  0x0000562c1ab4a281 in std::__1::__function::__func<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0, std::__1::allocator<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>, void ()>::operator()() (this=0x603000020440) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:354
#9  0x0000562c09f752c6 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x60400000e1d0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:507
#10 0x0000562c09f73fd9 in std::__1::function<void ()>::operator()() const (this=0x60400000e1d0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:1184
#11 0x0000562c1a5aafe9 in seastar::posix_thread::start_routine (arg=0x60400000e1d0) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
#12 0x00007f176dc0b14d in start_thread (arg=<optimized out>) at pthread_create.c:442
#13 0x00007f176dc8ca00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f17672c46c0 (LWP 53716)):
#0  __GI___libc_read (nbytes=8, buf=0x7f17672b4d00, fd=18) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=18, buf=0x7f17672b4d00, nbytes=8) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x0000562c09b02a00 in __interceptor_read () at /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1024
#3  0x0000562c1ab47387 in seastar::thread_pool::work (this=0x61d00002f880, name=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/thread_pool.cc:41
#4  0x0000562c1ab4f167 in seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0::operator()() const (this=0x603000023448) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/thread_pool.cc:29
#5  0x0000562c1ab4ef71 in std::__1::__invoke<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&>(seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&) (__f=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
#6  0x0000562c1ab4ee2d in std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&>(seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&) (__args=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:61
#7  0x0000562c1ab4ed95 in std::__1::__function::__alloc_func<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0, std::__1::allocator<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>, void ()>::operator()() (this=0x603000023448) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:180
#8  0x0000562c1ab4a281 in std::__1::__function::__func<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0, std::__1::allocator<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>, void ()>::operator()() (this=0x603000023440) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:354
#9  0x0000562c09f752c6 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x6040000121d0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:507
#10 0x0000562c09f73fd9 in std::__1::function<void ()>::operator()() const (this=0x6040000121d0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:1184
#11 0x0000562c1a5aafe9 in seastar::posix_thread::start_routine (arg=0x6040000121d0) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
#12 0x00007f176dc0b14d in start_thread (arg=<optimized out>) at pthread_create.c:442
#13 0x00007f176dc8ca00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f1767ad36c0 (LWP 53715)):
#0  __GI___libc_read (nbytes=8, buf=0x7f1767ac3d00, fd=15) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=15, buf=0x7f1767ac3d00, nbytes=8) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x0000562c09b02a00 in __interceptor_read () at /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1024
#3  0x0000562c1ab47387 in seastar::thread_pool::work (this=0x61d000010480, name=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/thread_pool.cc:41
#4  0x0000562c1ab4f167 in seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0::operator()() const (this=0x603000021c48) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/thread_pool.cc:29
#5  0x0000562c1ab4ef71 in std::__1::__invoke<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&>(seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&) (__f=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
#6  0x0000562c1ab4ee2d in std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&>(seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0&) (__args=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:61
#7  0x0000562c1ab4ed95 in std::__1::__function::__alloc_func<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0, std::__1::allocator<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>, void ()>::operator()() (this=0x603000021c48) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:180
#8  0x0000562c1ab4a281 in std::__1::__function::__func<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0, std::__1::allocator<seastar::thread_pool::thread_pool(seastar::reactor*, seastar::basic_sstring<char, unsigned int, 15u, true>)::$_0>, void ()>::operator()() (this=0x603000021c40) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:354
#9  0x0000562c09f752c6 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x6040000101d0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:507
#10 0x0000562c09f73fd9 in std::__1::function<void ()>::operator()() const (this=0x6040000101d0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:1184
#11 0x0000562c1a5aafe9 in seastar::posix_thread::start_routine (arg=0x6040000101d0) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
#12 0x00007f176dc0b14d in start_thread (arg=<optimized out>) at pthread_create.c:442
#13 0x00007f176dc8ca00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f17732c4480 (LWP 53710)):
#0  0x0000562c09eba8b1 in fmt::v8::basic_memory_buffer<char, 500ul, std::__1::allocator<char> >::basic_memory_buffer (this=0x7ffd66bb1620, alloc=...) at rp_deps_install/include/fmt/format.h:699
#1  0x0000562c0d81b712 in fmt::v8::detail::fallback_formatter<kafka::error_code, char, void>::format<fmt::v8::appender> (this=0x7ffd66bb15a0, value=@0x7ffd66bb9080: kafka::error_code::none, ctx=...) at rp_deps_install/include/fmt/ostream.h:91
#2  0x0000562c0d81b228 in fmt::v8::detail::value<fmt::v8::basic_format_context<fmt::v8::appender, char> >::format_custom_arg<kafka::error_code, fmt::v8::detail::fallback_formatter<kafka::error_code, char, void> > (arg=0x7ffd66bb9080, parse_ctx=..., ctx=...) at rp_deps_install/include/fmt/core.h:1282
#3  0x0000562c09ee1675 in fmt::v8::basic_format_arg<fmt::v8::basic_format_context<fmt::v8::appender, char> >::handle::format (this=0x7ffd66bb17c0, parse_ctx=..., ctx=...) at rp_deps_install/include/fmt/core.h:1558
#4  0x0000562c09e810fe in fmt::v8::detail::default_arg_formatter<char>::operator() (this=0x7ffd66bb1c10, h=...) at rp_deps_install/include/fmt/format.h:2202
#5  0x0000562c09ee44a9 in fmt::v8::visit_format_arg<fmt::v8::detail::default_arg_formatter<char>, fmt::v8::basic_format_context<fmt::v8::appender, char> > (vis=..., arg=...) at rp_deps_install/include/fmt/core.h:1622
#6  fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler::on_replacement_field(int, char const*) (this=0x7ffd66bb2ce0, id=0) at rp_deps_install/include/fmt/format.h:2962
#7  fmt::v8::detail::parse_replacement_field<char, fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler&>(char const*, char const*, fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler&) (begin=0x562bfb6d2b0e <str+14> "} mechanisms={}}}", end=0x562bfb6d2b1f <str+31> "", handler=...) at rp_deps_install/include/fmt/core.h:2591
#8  0x0000562c09e78867 in fmt::v8::detail::parse_format_string<false, char, fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler>(fmt::v8::basic_string_view<char>, fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler&&) (format_str=..., handler=...) at rp_deps_install/include/fmt/core.h:2626
#9  fmt::v8::detail::vformat_to<char> (buf=..., fmt=..., args=..., loc=...) at rp_deps_install/include/fmt/format.h:2988
#10 0x0000562c09f63f75 in fmt::v8::vprint<char> (os=..., format_str=..., args=...) at rp_deps_install/include/fmt/ostream.h:113
#11 0x0000562c111e2228 in fmt::v8::print<char [32], kafka::error_code const&, std::__1::vector<seastar::basic_sstring<char, unsigned int, 15u, true>, std::__1::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > > const&, char> (os=..., format_str=..., args=..., args=...) at rp_deps_install/include/fmt/ostream.h:130
#12 0x0000562c11072d20 in kafka::operator<< (o=..., v=...) at src/v/kafka/protocol/schemata/protocol.cc:3916
#13 0x0000562c0e0bc50d in kafka::operator<< (os=..., r=...) at /home/nwatkins/src/redpanda/src/v/kafka/protocol/sasl_handshake.h:66
#14 0x0000562c0e0bc130 in fmt::v8::detail::format_value<char, kafka::sasl_handshake_response> (buf=..., value=..., loc=...) at rp_deps_install/include/fmt/ostream.h:77
#15 0x0000562c0e0bb0bf in fmt::v8::detail::fallback_formatter<kafka::sasl_handshake_response, char, void>::format<fmt::v8::appender> (this=0x7ffd66bb4bc0, value=..., ctx=...) at rp_deps_install/include/fmt/ostream.h:92
#16 0x0000562c0e0baac8 in fmt::v8::detail::value<fmt::v8::basic_format_context<fmt::v8::appender, char> >::format_custom_arg<kafka::sasl_handshake_response, fmt::v8::detail::fallback_formatter<kafka::sasl_handshake_response, char, void> > (arg=0x7ffd66bb9080, parse_ctx=..., ctx=...) at rp_deps_install/include/fmt/core.h:1282
#17 0x0000562c09ee1675 in fmt::v8::basic_format_arg<fmt::v8::basic_format_context<fmt::v8::appender, char> >::handle::format (this=0x7ffd66bb4de0, parse_ctx=..., ctx=...) at rp_deps_install/include/fmt/core.h:1558
#18 0x0000562c09e810fe in fmt::v8::detail::default_arg_formatter<char>::operator() (this=0x7ffd66bb5230, h=...) at rp_deps_install/include/fmt/format.h:2202
#19 0x0000562c09ee44a9 in fmt::v8::visit_format_arg<fmt::v8::detail::default_arg_formatter<char>, fmt::v8::basic_format_context<fmt::v8::appender, char> > (vis=..., arg=...) at rp_deps_install/include/fmt/core.h:1622
#20 fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler::on_replacement_field(int, char const*) (this=0x7ffd66bb6300, id=4) at rp_deps_install/include/fmt/format.h:2962
#21 fmt::v8::detail::parse_replacement_field<char, fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler&>(char const*, char const*, fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler&) (begin=0x562bfb640648 <str+40> "}", end=0x562bfb640649 <str+41> "", handler=...) at rp_deps_install/include/fmt/core.h:2591
#22 0x0000562c09e78d56 in fmt::v8::detail::parse_format_string<false, char, fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler>(fmt::v8::basic_string_view<char>, fmt::v8::detail::vformat_to<char>(fmt::v8::detail::buffer<char>&, fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<std::__1::conditional<std::is_same<fmt::v8::type_identity<char>::type, char>::value, fmt::v8::appender, std::__1::back_insert_iterator<fmt::v8::detail::buffer<fmt::v8::type_identity<char>::type> > >::type, fmt::v8::type_identity<char>::type> >, fmt::v8::detail::locale_ref)::format_handler&&) (format_str=..., handler=...) at rp_deps_install/include/fmt/core.h:2660
#23 fmt::v8::detail::vformat_to<char> (buf=..., fmt=..., args=..., loc=...) at rp_deps_install/include/fmt/format.h:2988
#24 0x0000562c09e7469a in fmt::v8::vformat_to<seastar::internal::log_buf::inserter_iterator, 0> (out=..., fmt=..., args=...) at rp_deps_install/include/fmt/core.h:3128
#25 0x0000562c10fa6b5f in fmt::v8::format_to<seastar::internal::log_buf::inserter_iterator, char const*, int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, char const* const&, kafka::sasl_handshake_response&, 0> (out=..., fmt=..., args=..., args=..., args=..., args=..., args=...) at rp_deps_install/include/fmt/core.h:3148
#26 seastar::logger::log<char const*, int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, char const* const&, kafka::sasl_handshake_response&>(seastar::log_level, seastar::logger::format_info, char const*&&, int&&, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, char const* const&, kafka::sasl_handshake_response&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const (this=0x7ffd66bb8858, it=...) at rp_deps_install/include/seastar/util/log.hh:222
#27 0x0000562c10fa5bba in seastar::logger::lambda_log_writer<seastar::logger::log<char const*, int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, char const* const&, kafka::sasl_handshake_response&>(seastar::log_level, seastar::logger::format_info, char const*&&, int&&, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, char const* const&, kafka::sasl_handshake_response&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (this=0x7ffd66bb8850, it=...) at rp_deps_install/include/seastar/util/log.hh:108
#28 0x0000562c1b715c44 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::$_0::operator()(seastar::internal::log_buf::inserter_iterator) const (this=0x7ffd66bb7f60, it=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/util/log.cc:323
#29 0x0000562c1b71371c in seastar::logger::do_log (this=0x562c2d2972d8 <kafka::client::kclog>, level=seastar::log_level::debug, writer=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/util/log.cc:335
#30 0x0000562c10fa551d in seastar::logger::log<char const*, int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, char const* const&, kafka::sasl_handshake_response&> (this=0x562c2d2972d8 <kafka::client::kclog>, level=seastar::log_level::debug, fmt=..., args=..., args=..., args=..., args=..., args=...) at rp_deps_install/include/seastar/util/log.hh:227
#31 0x0000562c10fa4ec6 in seastar::logger::debug<char const*, int, detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >&, char const* const&, kafka::sasl_handshake_response&> (this=0x562c2d2972d8 <kafka::client::kclog>, fmt=..., args=..., args=..., args=..., args=..., args=...) at rp_deps_install/include/seastar/util/log.hh:370
#32 0x0000562c10fa47b1 in kafka::client::broker::dispatch<kafka::sasl_handshake_request, kafka::sasl_handshake_response>(kafka::sasl_handshake_request)::{lambda()#1}::operator()()::{lambda(kafka::sasl_handshake_response)#1}::operator()(kafka::sasl_handshake_response) const (this=0x60d000511668, res=...) at /home/nwatkins/src/redpanda/src/v/kafka/client/broker.h:70
#33 0x0000562c10fa4091 in seastar::futurize<kafka::sasl_handshake_response>::invoke<kafka::client::broker::dispatch<kafka::sasl_handshake_request, kafka::sasl_handshake_response>(kafka::sasl_handshake_request)::{lambda()#1}::operator()()::{lambda(kafka::sasl_handshake_response)#1}&, kafka::sasl_handshake_response>(kafka::client::broker::dispatch<kafka::sasl_handshake_request, kafka::sasl_handshake_response>(kafka::sasl_handshake_request)::{lambda()#1}::operator()()::{lambda(kafka::sasl_handshake_response)#1}&, kafka::sasl_handshake_response&&) (func=..., args=...) at rp_deps_install/include/seastar/core/future.hh:2151
#34 0x0000562c10fa3e58 in seastar::futurize_invoke<kafka::client::broker::dispatch<kafka::sasl_handshake_request, kafka::sasl_handshake_response>(kafka::sasl_handshake_request)::{lambda()#1}::operator()()::{lambda(kafka::sasl_handshake_response)#1}&, kafka::sasl_handshake_response>(kafka::client::broker::dispatch<kafka::sasl_handshake_request, kafka::sasl_handshake_response>(kafka::sasl_handshake_request)::{lambda()#1}::operator()()::{lambda(kafka::sasl_handshake_response)#1}&, kafka::sasl_handshake_response&&) (func=..., args=...) at rp_deps_install/include/seastar/core/future.hh:2180
#35 0x0000562c10fa3cd6 in _ZZN7seastar6futureIN5kafka23sasl_handshake_responseEE4thenIZZNS1_6client6broker8dispatchINS1_22sasl_handshake_requestES2_EENS0_IT0_EET_ENUlvE_clEvEUlS2_E_S3_EES9_OSB_ENUlDpOT_E_clIJS2_EEEDaSH_ (this=0x60d000511668, args=...) at rp_deps_install/include/seastar/core/future.hh:1541
#36 0x0000562c10fa3bd4 in _ZN7seastar20noncopyable_functionIFNS_6futureIN5kafka23sasl_handshake_responseEEEOS3_EE17direct_vtable_forIZNS4_4thenIZZNS2_6client6broker8dispatchINS2_22sasl_handshake_requestES3_EENS1_IT0_EET_ENUlvE_clEvEUlS3_E_S4_EESE_OSG_EUlDpOT_E_E4callEPKS7_S5_ (func=0x60d000511668, args=...) at rp_deps_install/include/seastar/util/noncopyable_function.hh:124
#37 0x0000562c10fac4d4 in seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>::operator()(kafka::sasl_handshake_response&&) const (this=0x60d000511668, args=...) at rp_deps_install/include/seastar/util/noncopyable_function.hh:209
#38 0x0000562c10fac2e0 in std::__1::__invoke<seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, kafka::sasl_handshake_response>(seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, kafka::sasl_handshake_response&&) (__f=..., __args=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
#39 0x0000562c10fac1c8 in std::__1::invoke<seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, kafka::sasl_handshake_response>(seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, kafka::sasl_handshake_response&&) (__f=..., __args=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:93
#40 0x0000562c10fabf28 in seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, kafka::sasl_handshake_response>(seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, kafka::sasl_handshake_response&&) (func=..., v=...) at rp_deps_install/include/seastar/core/future.hh:1225
#41 0x0000562c10fabe02 in seastar::future<kafka::sasl_handshake_response>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>, seastar::future<kafka::sasl_handshake_response> >(seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&&)::{lambda(seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>&&, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, seastar::future_state<kafka::sasl_handshake_response>&&)#1}::operator()(seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>&&, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, seastar::future_state<kafka::sasl_handshake_response>&&) const::{lambda()#1}::operator()() const (this=0x7ffd66bb9660) at rp_deps_install/include/seastar/core/future.hh:1596
#42 0x0000562c10fabb31 in seastar::futurize<seastar::future<kafka::sasl_handshake_response> >::satisfy_with_result_of<seastar::future<kafka::sasl_handshake_response>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>, seastar::future<kafka::sasl_handshake_response> >(seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&&)::{lambda(seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>&&, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, seastar::future_state<kafka::sasl_handshake_response>&&)#1}::operator()(seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>&&, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, seastar::future_state<kafka::sasl_handshake_response>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>&&, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&&) (pr=..., func=...) at rp_deps_install/include/seastar/core/future.hh:2134
#43 0x0000562c10fab907 in seastar::future<kafka::sasl_handshake_response>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>, seastar::future<kafka::sasl_handshake_response> >(seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&&)::{lambda(seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>&&, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, seastar::future_state<kafka::sasl_handshake_response>&&)#1}::operator()(seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>&&, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, seastar::future_state<kafka::sasl_handshake_response>&&) const (this=0x60d000511600, pr=..., func=..., state=...) at rp_deps_install/include/seastar/core/future.hh:1589
#44 0x0000562c10fab09b in seastar::continuation<seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>, seastar::future<kafka::sasl_handshake_response>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>, seastar::future<kafka::sasl_handshake_response> >(seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&&)::{lambda(seastar::internal::promise_base_with_type<kafka::sasl_handshake_response>&&, seastar::noncopyable_function<seastar::future<kafka::sasl_handshake_response> (kafka::sasl_handshake_response&&)>&, seastar::future_state<kafka::sasl_handshake_response>&&)#1}, kafka::sasl_handshake_response>::run_and_dispose() (this=0x60d000511600) at rp_deps_install/include/seastar/core/future.hh:781
#45 0x0000562c1a70413e in seastar::reactor::run_tasks (this=0x623000000100, tq=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
#46 0x0000562c1a709f9f in seastar::reactor::run_some_tasks (this=0x623000000100) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
#47 0x0000562c1a70eb00 in seastar::reactor::do_run (this=0x623000000100) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
#48 0x0000562c1a70c689 in seastar::reactor::run (this=0x623000000100) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2789
#49 0x0000562c1a3fc17d in seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) (this=0x7ffd66bbbd00, ac=13, av=0x7ffd66bc02e8, func=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:265
#50 0x0000562c1a3f9836 in seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) (this=0x7ffd66bbbd00, ac=13, av=0x7ffd66bc02e8, func=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:156
#51 0x0000562c09bea165 in application::run (this=0x7ffd66bbde40, ac=13, av=0x7ffd66bc02e8) at /home/nwatkins/src/redpanda/src/v/redpanda/application.cc:304
#52 0x0000562c09ba79d7 in main (argc=13, argv=0x7ffd66bc02e8) at /home/nwatkins/src/redpanda/src/v/redpanda/main.cc:22

Thread 3 (Thread 0x7f176a33e6c0 (LWP 53711)):
#0  0x0000562c09b69750 in __asan_memcpy () at /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:22
#1  0x00007f176d8debed in nettle_sha256_update (ctx=0x617001c58e60, length=32, data=0x7f176a32c0c0 "W\253\314\004\244\366v_") at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle/sha256.c:100
#2  0x00007f176f137f44 in x86_sha256_update (ctx=0x617001c58e60, length=0, data=0x7f176a32c0c0 "W\253\314\004\244\366v_") at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/accelerated/x86/sha-x86-ssse3.c:177
#3  0x00007f176d8a16a7 in nettle_hmac_digest (outer=0x617001c58d80, inner=0x617001c58df0, state=0x617001c58e60, hash=0x7f176f261a40 <x86_sha256>, length=32, dst=0x7f176a32c5f0 "\276%\217\333=\002\244AE\321\343f\003p4\222\v\fV\337\372\061\226GU7\346\344m9\342i@\305\062j\027\177") at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/nettle-prefix/src/nettle/hmac.c:113
#4  0x00007f176f128adb in x86_hmac_sha256_digest (ctx=0x617001c58d80, length=32, digest=0x7f176a32c5f0 "\276%\217\333=\002\244AE\321\343f\003p4\222\v\fV\337\372\061\226GU7\346\344m9\342i@\305\062j\027\177") at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/accelerated/x86/hmac-x86-ssse3.c:99
#5  0x00007f176f126838 in wrap_x86_hmac_output (src_ctx=0x617001c58d80, digest=0x7f176a32c5f0, digestsize=32) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/accelerated/x86/hmac-x86-ssse3.c:271
#6  0x00007f176eca7f27 in _gnutls_mac_output (handle=0x60600054c680, digest=0x7f176a32c5f0) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/hash_int.h:97
#7  0x00007f176eca7d0d in gnutls_hmac_output (handle=0x60600054c680, digest=0x7f176a32c5f0) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/crypto-api.c:433
#8  0x0000562c0be71be6 in internal::hmac<(gnutls_mac_algorithm_t)6, 32ul>::reset (this=0x7f176a32c4c0) at /home/nwatkins/src/redpanda/src/v/hashing/secure.h:76
#9  0x0000562c0be7138d in security::scram_algorithm<internal::hmac<(gnutls_mac_algorithm_t)6, 32ul>, internal::hash<(gnutls_digest_algorithm_t)6, 32ul>, 130ul, 4096>::hi (str=..., salt=..., iterations=4096) at /home/nwatkins/src/redpanda/src/v/security/scram_algorithm.h:291
#10 0x0000562c10f66256 in kafka::client::do_authenticate_scram<security::scram_algorithm<internal::hmac<(gnutls_mac_algorithm_t)6, 32ul>, internal::hash<(gnutls_digest_algorithm_t)6, 32ul>, 130ul, 4096> > (broker=..., username=..., password=...) at /home/nwatkins/src/redpanda/src/v/kafka/client/sasl_client.cc:159
#11 0x0000562c0a07cd01 in std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume (this=0x7f176a32d2e0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168
#12 0x0000562c0a07c826 in seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose (this=0x61800045bc90) at rp_deps_install/include/seastar/core/coroutine.hh:120
#13 0x0000562c1a70413e in seastar::reactor::run_tasks (this=0x62300000fd00, tq=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
#14 0x0000562c1a709f9f in seastar::reactor::run_some_tasks (this=0x62300000fd00) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
#15 0x0000562c1a70eb00 in seastar::reactor::do_run (this=0x62300000fd00) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
#16 0x0000562c1a87d00b in seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92::operator()() const (this=0x611000008108) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4140
#17 0x0000562c1a87bbf1 in std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92&) (__f=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
#18 0x0000562c1a87baad in std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92&) (__args=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:61
#19 0x0000562c1a87ba15 in std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92>, void ()>::operator()() (this=0x611000008108) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:180
#20 0x0000562c1a876a41 in std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92>, void ()>::operator()() (this=0x611000008100) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:354
#21 0x0000562c09f752c6 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x60400000c350) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:507
#22 0x0000562c09f73fd9 in std::__1::function<void ()>::operator()() const (this=0x60400000c350) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:1184
#23 0x0000562c1a5aafe9 in seastar::posix_thread::start_routine (arg=0x60400000c350) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
#24 0x00007f176dc0b14d in start_thread (arg=<optimized out>) at pthread_create.c:442
#25 0x00007f176dc8ca00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f17693206c0 (LWP 53713)):
#0  0x00007f176f126431 in wrap_x86_hmac_update (_ctx=0x0, text=0x606000543fe0, textsize=139738525787504) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/accelerated/x86/hmac-x86-ssse3.c:252
#1  0x00007f176eca7cd3 in _gnutls_mac (handle=0x606000543fe0, text=0x7f176930e570, textlen=32) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/hash_int.h:89
#2  0x00007f176eca7b75 in gnutls_hmac (handle=0x606000543fe0, ptext=0x7f176930e570, ptext_len=32) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/crypto-api.c:418
#3  0x0000562c0be7246a in internal::hmac<(gnutls_mac_algorithm_t)6, 32ul>::update (this=0x7f176930e4c0, data=0x7f176930e570, size=32) at /home/nwatkins/src/redpanda/src/v/hashing/secure.h:96
#4  0x0000562c0be71c4e in internal::hmac<(gnutls_mac_algorithm_t)6, 32ul>::update<32ul> (this=0x7f176930e4c0, data=...) at /home/nwatkins/src/redpanda/src/v/hashing/secure.h:68
#5  0x0000562c0be71367 in security::scram_algorithm<internal::hmac<(gnutls_mac_algorithm_t)6, 32ul>, internal::hash<(gnutls_digest_algorithm_t)6, 32ul>, 130ul, 4096>::hi (str=..., salt=..., iterations=4096) at /home/nwatkins/src/redpanda/src/v/security/scram_algorithm.h:290
#6  0x0000562c10f66256 in kafka::client::do_authenticate_scram<security::scram_algorithm<internal::hmac<(gnutls_mac_algorithm_t)6, 32ul>, internal::hash<(gnutls_digest_algorithm_t)6, 32ul>, 130ul, 4096> > (broker=..., username=..., password=...) at /home/nwatkins/src/redpanda/src/v/kafka/client/sasl_client.cc:159
#7  0x0000562c0a07cd01 in std::__1::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume (this=0x7f176930f2e0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__coroutine/coroutine_handle.h:168
#8  0x0000562c0a07c826 in seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose (this=0x618000465890) at rp_deps_install/include/seastar/core/coroutine.hh:120
#9  0x0000562c1a70413e in seastar::reactor::run_tasks (this=0x62300002f500, tq=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2330
#10 0x0000562c1a709f9f in seastar::reactor::run_some_tasks (this=0x62300002f500) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2737
#11 0x0000562c1a70eb00 in seastar::reactor::do_run (this=0x62300002f500) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2906
#12 0x0000562c1a87d00b in seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92::operator()() const (this=0x611000008388) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:4140
#13 0x0000562c1a87bbf1 in std::__1::__invoke<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92&) (__f=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/type_traits:3640
#14 0x0000562c1a87baad in std::__1::__invoke_void_return_wrapper<void, true>::__call<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92&) (__args=...) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/invoke.h:61
#15 0x0000562c1a87ba15 in std::__1::__function::__alloc_func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92>, void ()>::operator()() (this=0x611000008388) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:180
#16 0x0000562c1a876a41 in std::__1::__function::__func<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92, std::__1::allocator<seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_92>, void ()>::operator()() (this=0x611000008380) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:354
#17 0x0000562c09f752c6 in std::__1::__function::__value_func<void ()>::operator()() const (this=0x60400000c3d0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:507
#18 0x0000562c09f73fd9 in std::__1::function<void ()>::operator()() const (this=0x60400000c3d0) at /home/nwatkins/src/redpanda/vbuild/llvm/install/bin/../include/c++/v1/__functional/function.h:1184
#19 0x0000562c1a5aafe9 in seastar::posix_thread::start_routine (arg=0x60400000c3d0) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:73
#20 0x00007f176dc0b14d in start_thread (arg=<optimized out>) at pthread_create.c:442
#21 0x00007f176dc8ca00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f1769b2f6c0 (LWP 53712)):
#0  _Unwind_GetPtr (index=16, context=0x7f1769b18b80) at ../../../libgcc/unwind-dw2.c:251
#1  uw_update_context (fs=0x7f1769b18c70, context=0x7f1769b18b80) at ../../../libgcc/unwind-dw2.c:1546
#2  _Unwind_Backtrace (trace=0x7f176dc98d40 <backtrace_helper>, trace_argument=0x7f1769b18e30) at ../../../libgcc/unwind.inc:326
#3  0x00007f176dc98e43 in __GI___backtrace (array=<optimized out>, size=100) at backtrace.c:78
#4  0x0000562c09b2303b in __interceptor_backtrace () at /home/nwatkins/src/redpanda/vbuild/llvm/src/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:4277
#5  0x0000562c1a908815 in seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}&&) (func=...) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
#6  0x0000562c1a907ced in seastar::backtrace_buffer::append_backtrace_oneline (this=0x7f1769b19da0) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:768
#7  0x0000562c1a6e3ff4 in seastar::print_with_backtrace (buf=..., oneline=true) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:787
#8  0x0000562c1a6de165 in seastar::internal::cpu_stall_detector::generate_trace (this=0x61100000ff40) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1346
#9  0x0000562c1a6ddc08 in seastar::internal::cpu_stall_detector::maybe_report (this=0x61100000ff40) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1088
#10 0x0000562c1a6de3c9 in seastar::internal::cpu_stall_detector::on_signal (this=0x61100000ff40) at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1105
#11 0x0000562c1a6e3e3b in seastar::reactor::block_notifier () at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1329
#12 <signal handler called>
#13 sha256_block_data_order_avx2 () at /home/nwatkins/src/redpanda/vbuild/debug/clang/v_deps_build/gnutls-prefix/src/gnutls/lib/accelerated/x86/elf/sha256-ssse3-x86_64.s:4597
Backtrace stopped: Cannot access memory at address 0x8095f88982a5b4e

quit

dotnwat · 2022-12-18T20:07:43Z

See https://gitlab.com/gnutls/gnutls/-/issues/1111

And some release notes from Scylla

Stability: gnutls 3.6.x causes the unwinder to crash when collecting a backtrace. This can crash a node when the stall detector detects a stall, and that stall happens to be in gnutls. Fix by pulling in gnutls 3.7.2 from Fedora 34.

Lazin added kind/bug Something isn't working area/schema-registry Schema Registry service within Redpanda ci-failure labels Oct 24, 2022

Lazin changed the title ~~Segfault in SchemaRegistryBasicAuthTest.test_delete_subject~~ Segfault in SchemaRegistryBasicAuthTest.test_delete_subject Oct 24, 2022

dotnwat assigned NyaliaLui and dotnwat Oct 26, 2022

NyaliaLui mentioned this issue Oct 26, 2022

tests: enable debug logging for pandaproxy #6958

Closed

6 tasks

rystsov mentioned this issue Oct 26, 2022

Segfault in SchemaRegistryAutoAuthTest.test_get_subjects #6961

Closed

mmedenjak added the area/security label Oct 31, 2022

BenPope mentioned this issue Nov 1, 2022

Failure in SchemaRegistryBasicAuthTest.test_delete_subject #6944

Closed

piyushredpanda unassigned dotnwat Nov 2, 2022

BenPope added a commit to BenPope/redpanda that referenced this issue Nov 2, 2022

pp: Wrap inform in coroutine::lambda

677ba09

Fix redpanda-data#6903 Signed-off-by: Ben Pope <ben@redpanda.com>

BenPope added a commit to BenPope/redpanda that referenced this issue Nov 2, 2022

pp: Wrap inform in coroutine::lambda

b46f8fb

Fix redpanda-data#6903 Signed-off-by: Ben Pope <ben@redpanda.com>

piyushredpanda assigned BenPope and unassigned NyaliaLui Nov 2, 2022

dlex mentioned this issue Nov 9, 2022

Lower log error severity to WARN for errc::shutting_down in k/group #7105

Merged

6 tasks

BenPope added the sev/high loss of availability, pathological performance degradation, recoverable corruption label Nov 9, 2022

BenPope mentioned this issue Nov 10, 2022

Segfault in SchemaRegistryAutoAuthTest.test_protobuf #7216

Closed

abhijat mentioned this issue Nov 11, 2022

segmentation violation during rptest.tests.pandaproxy_test.BasicAuthScaleTest.test_many_users.num_users=500 #7247

Closed

dotnwat self-assigned this Nov 16, 2022

NyaliaLui self-assigned this Nov 16, 2022

graphcareful mentioned this issue Nov 16, 2022

controller: fix Redpanda 22.3 failing to start when license was loaded with 22.2 Redpanda #7312

Merged

6 tasks

dlex mentioned this issue Nov 21, 2022

Fix a use-after-move in partition_balancer #7406

Merged

6 tasks

NyaliaLui mentioned this issue Nov 23, 2022

segv rptest.tests.pandaproxy_test.PandaProxyAutoAuthTest.test_subscribe_consumer_validation #7383

Closed

piyushredpanda unassigned dotnwat and NyaliaLui Nov 23, 2022

BenPope assigned dotnwat and unassigned BenPope Nov 23, 2022

jcsp changed the title ~~Segfault in SchemaRegistryBasicAuthTest.test_delete_subject~~ Segfault in SchemaRegistryBasicAuthTest.test_delete_subject, BasicAuthScaleTest.test_many_users Nov 23, 2022

ballard26 mentioned this issue Dec 1, 2022

Add an asychronous zstd stream compression class #5565

Merged

jcsp mentioned this issue Dec 2, 2022

cloud_storage: fix uploads after topic deletion (refactor archivers to be owned by cluster::partition) #7547

Merged

6 tasks

jcsp closed this as completed Dec 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segfault in `SchemaRegistryBasicAuthTest`.`test_delete_subject`, `BasicAuthScaleTest`.`test_many_users` #6903

Segfault in `SchemaRegistryBasicAuthTest`.`test_delete_subject`, `BasicAuthScaleTest`.`test_many_users` #6903

Lazin commented Oct 24, 2022 •

edited by BenPope

Loading

NyaliaLui commented Oct 26, 2022

BenPope commented Oct 27, 2022

piyushredpanda commented Nov 2, 2022

NyaliaLui commented Nov 2, 2022 •

edited

Loading

BenPope commented Nov 2, 2022

dlex commented Nov 9, 2022

BenPope commented Nov 10, 2022 •

edited

Loading

abhijat commented Nov 14, 2022 •

edited

Loading

NyaliaLui commented Nov 14, 2022

andijcr commented Nov 14, 2022

graphcareful commented Nov 16, 2022

dlex commented Nov 21, 2022

piyushredpanda commented Nov 23, 2022

jcsp commented Nov 23, 2022

mmaslankaprv commented Nov 30, 2022

ballard26 commented Dec 1, 2022

jcsp commented Dec 2, 2022

jcsp commented Dec 12, 2022

dotnwat commented Dec 16, 2022

dotnwat commented Dec 16, 2022

dotnwat commented Dec 18, 2022 •

edited

Loading

dotnwat commented Dec 18, 2022

Segfault in SchemaRegistryBasicAuthTest.test_delete_subject, BasicAuthScaleTest.test_many_users #6903

Segfault in SchemaRegistryBasicAuthTest.test_delete_subject, BasicAuthScaleTest.test_many_users #6903

Comments

Lazin commented Oct 24, 2022 • edited by BenPope Loading

Version & Environment

NyaliaLui commented Oct 26, 2022

BenPope commented Oct 27, 2022

piyushredpanda commented Nov 2, 2022

NyaliaLui commented Nov 2, 2022 • edited Loading

BenPope commented Nov 2, 2022

dlex commented Nov 9, 2022

BenPope commented Nov 10, 2022 • edited Loading

abhijat commented Nov 14, 2022 • edited Loading

NyaliaLui commented Nov 14, 2022

andijcr commented Nov 14, 2022

graphcareful commented Nov 16, 2022

dlex commented Nov 21, 2022

piyushredpanda commented Nov 23, 2022

jcsp commented Nov 23, 2022

mmaslankaprv commented Nov 30, 2022

ballard26 commented Dec 1, 2022

jcsp commented Dec 2, 2022

jcsp commented Dec 12, 2022

dotnwat commented Dec 16, 2022

dotnwat commented Dec 16, 2022

dotnwat commented Dec 18, 2022 • edited Loading

dotnwat commented Dec 18, 2022

Segfault in `SchemaRegistryBasicAuthTest`.`test_delete_subject`, `BasicAuthScaleTest`.`test_many_users` #6903

Segfault in `SchemaRegistryBasicAuthTest`.`test_delete_subject`, `BasicAuthScaleTest`.`test_many_users` #6903

Lazin commented Oct 24, 2022 •

edited by BenPope

Loading

NyaliaLui commented Nov 2, 2022 •

edited

Loading

BenPope commented Nov 10, 2022 •

edited

Loading

abhijat commented Nov 14, 2022 •

edited

Loading

dotnwat commented Dec 18, 2022 •

edited

Loading