[WIP] Fix flaky tests

[kciesielski]

No description provided.

[kciesielski]

Issue still happens on Java 21

2024-06-07T10:54:12.5578064Z [TESTINFO] Test sttp.tapir.server.vertx.zio.VertxZioServerStubTest PASSED in 3 ms,601138
2024-06-07T10:54:12.5578953Z [TESTINFO] Finished test group: sttp.tapir.server.vertx.zio.VertxZioServerStubTest result: Passed,601138
2024-06-07T10:54:12.5579823Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[36mRun completed in 33 seconds, 940 milliseconds.�[0m�[0m
2024-06-07T10:54:12.5580497Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[36mTotal number of tests run: 271�[0m�[0m
2024-06-07T10:54:12.5581099Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[36mSuites: completed 4, aborted 0�[0m�[0m
2024-06-07T10:54:12.5581851Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[36mTests: succeeded 271, failed 0, canceled 0, ignored 0, pending 0�[0m�[0m
2024-06-07T10:54:12.5582541Z �[0m[�[0m�[0minfo�[0m] �[0m�[0m�[32mAll tests passed.�[0m�[0m
2024-06-07T11:06:20.2099155Z �[34mclientTestServer�[0m ... killing ...
2024-06-07T11:06:30.2053475Z ##[warning]Attempt 1 failed. Reason: Timeout of 900000ms hit

I'm curious about this line:

2024-06-07T11:06:20.2099155Z �[34mclientTestServer�[0m ... killing ...

it's the last line logged, and it is logged after a long gap. Then, timeout trigger kicks in.

[kciesielski]

I found one problematic test - "empty client stream" in ServerWebSocketTests. It fails pretty frequently for VertX. The freezes aren't probably directly related, but fixing it should help a bit anyway.

[kciesielski]

Spotted tests that fail sometimes, fixing them in a separate PR
#3847

[kciesielski]

Disabling tests for all zio-related backends made the issue go away.
Restoring http4s-zio made it reappear. I saw a lot of exceptions in logs on closing WebSockets by http4s-zio, so I disabled only WS tests for this backend. It didn't help, after a few runs another timeout occured.
What's noteworthy, logs from this run show that all zio-http4s related tests pass. However, there seems something to be happening with verx-cats. There's a "starting" entry logged:

[TESTINFO] Starting test group: sttp.tapir.server.vertx.cats.VertxCatsServerStubStreamingTest

but no corresponding [TESTINFO] Finished test group: entry. There are no logs for this backend at all.

[adamw]

Did you find that separating the ZIO tests into separate build steps increases the number of builds that pass? I'm thinking of introducing only this change for now, even if it doesn't solve the issue, maybe it will improve our builds nonetheless

[kciesielski]

This separation shows that it's clearly the zio part that fails with freezes.
In combination with pong issues #3944 these failures often exhaust retries.