Remove aarch64Alpine from default jdk11u pipeline config due to test hangs reliability #349

andrew-m-leonard · 2022-06-29T13:33:43Z

Due to nearly always hanging during jdk11u test jobs, the aarch64Alpine platform is being removed from the default nightly jdk11u pipeline config.
See: adoptium/aqa-tests#3799

Signed-off-by: Andrew Leonard anleonar@redhat.com

karianna · 2022-06-29T13:41:20Z

Do we understand why? Was this a platform we intended to release in July PSU?

github-actions · 2022-06-29T13:56:15Z

Thank you for creating a pull request!

Please check out the information below if you have not made a pull request here before (or if you need a reminder how things work).

Code Quality and Contributing Guidelines

If you have not done so already, please familiarise yourself with our Contributing Guidelines and Code Of Conduct, even if you have contributed before.

Tests

Github actions will run a set of jobs against your PR that will lint and unit test your changes. Keep an eye out for the results from these on the latest commit you submitted. For more information, please see our testing documentation.

In order to run the advanced pipeline tests (executing a set of mock pipelines), I require an admin to post run tests on this PR.
If you are not an admin, please ask for one's attention in #infrastructure on Slack or ping one here.

andrew-m-leonard · 2022-06-29T14:40:10Z

Do we understand why? Was this a platform we intended to release in July PSU?

@karianna aarch64Alpine is not for July
We don't understand why currently, it's being consistently hanging during tests in different locations, @Haroon-Khel logged on to check the hanging processes, but they just showed the processes in a "Wait" state.

sxa · 2022-06-30T08:39:28Z

@andrew-m-leonard I thought this was only affecting JDK11? Have the problems been seen on other version?

andrew-m-leonard · 2022-06-30T08:45:29Z

@sxa yes, seen on jdk17u as well.

I'm just looking at the current running last night's tests, and so far they are all still running.... so wondering if something may have fixed it...?
Interesting all our arm nodes have also been out, and also Equinix's Ed's comment maybe? https://adoptium.slack.com/archives/C53GHCXL4/p1656526042724669

sxa · 2022-06-30T09:42:08Z

jdk17u

Got a link? https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-alpine-linux-aarch64-temurin/ doesn't look like it's experienced any hangs on a initial look

andrew-m-leonard · 2022-06-30T09:55:40Z

jdk17u

Got a link? https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-alpine-linux-aarch64-temurin/ doesn't look like it's experienced any hangs on a initial look

adoptium/aqa-tests#3806

andrew-m-leonard · 2022-06-30T10:04:31Z

run tests

sxa · 2022-06-30T10:14:43Z

jdk17u

Got a link? https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-alpine-linux-aarch64-temurin/ doesn't look like it's experienced any hangs on a initial look

adoptium/aqa-tests#3806

I'm not sure what I'm reading in there. That is in relation to pipeline 34 which appearred to fail somewhere in the GPG signing step (that job has now been deleted so I can't look into it).

There was a comment which links to your issue adoptium/aqa-tests#3799

Are you sure that any delay in that pipeline wasn't just a hold up caused by the executors being held up by JDK11 jobs?

All I can see from pipeline 34's subjobs (NOTE: It's a weekly pipeline so would not nceessarily be directly comparable to the others) was https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux_testList_1/ which you killed #3 from even though it looks like it hadn't got to the end (NOTE that that the previous runs of that job took longer than the 4h4 it was at when it was terminated)

andrew-m-leonard · 2022-06-30T11:01:11Z

You maybe right being only a jdk11u, although jdk19 Smoke test has hung this morning
The above jdk17u hang I thought @sophia-guo had seen during triage?

sxa · 2022-06-30T11:04:14Z

You maybe right being only a jdk11u, although jdk19 Smoke test has hung this morning

To be 100% clear, that was a smoke test hang on x64, not aarch64, so not relevant to this PR and from the look of it it's not new as it's hit the timeout in all runs visible on https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk19/job/jdk19-alpine-linux-x64-temurin_SmokeTests/

andrew-m-leonard · 2022-06-30T11:05:51Z

You maybe right being only a jdk11u, although jdk19 Smoke test has hung this morning

To be 100% clear, that was a smoke test hang on x64, not aarch64, so not relevant to this PR.

Not sure we can totally rule that out, both Alpine

eclipse-temurin-bot · 2022-06-30T11:28:23Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

sxa · 2022-06-30T12:34:28Z

Not sure we can totally rule that out, both Alpine

Not totally, but JDK19 on x64 has had a 100% hang rate recently, aarch64 has not failed recently.

sophia-guo · 2022-06-30T18:41:46Z

You maybe right being only a jdk11u, although jdk19 Smoke test has hung this morning The above jdk17u hang I thought @sophia-guo had seen during triage?

Yes, I think it is same for jdk17, test jobs are timeout and abort.
Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux ❌ ABORTED ❌

Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux_testList_0 ❌ ABORTED ❌
jdk_security3_2 => deep history 0/1 passed | possible issues |

Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux_testList_1 ❌ ABORTED ❌
jdk_security4_0 => deep history 0/1 passed | possible issues |

sxa · 2022-07-01T06:45:01Z

You maybe right being only a jdk11u, although jdk19 Smoke test has hung this morning The above jdk17u hang I thought @sophia-guo had seen during triage?

Yes, I think it is same for jdk17, test jobs are timeout and abort. Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux ❌ ABORTED ❌

Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux_testList_0 ❌ ABORTED ❌ jdk_security3_2 => deep history 0/1 passed | possible issues |

Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux_testList_1 ❌ ABORTED ❌ jdk_security4_0 => deep history 0/1 passed | possible issues |

@sophia-guo Why do you say they timed out? The first of those was the top level job that had the two testList ones underneath it. The first testList0 was stopped by Andrew after 4h03m when it typically takes between 4-5 hours so it did not timeout, but was stopped before it got to the normal time it would take to complete:

15:40:30  XML output with verification to /home/jenkins/workspace/Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux_testList_0/aqa-tests/TKG/output_16563194353924/jdk_security3_2/work
Aborted by [Andrew Leonard](https://ci.adoptopenjdk.net/user/andrew-m-leonard)
15:46:27  Sending interrupt signal to process
15:46:31  Terminated

The testlList_1 job was the same - killed earlier than the amount of time it would normally take to run.

15:45:29  XML output with verification to /home/jenkins/workspace/Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux_testList_1/aqa-tests/TKG/output_16563193343027/jdk_security4_0/work
Aborted by [Andrew Leonard](https://ci.adoptopenjdk.net/user/andrew-m-leonard)
15:46:18  Sending interrupt signal to process
15:46:20  Terminated

As a result of those two being aborted, the top level one was marked as aborted too.

As I said earlier, I haven't seen any evidence that JDK17/alpine/aarch64 has experienced any unusual hangs, only forced aborts.

https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_extended.openjdk_aarch64_alpine-linux_testList_1/1/ may have hit a timeout, but (a) there were a lot of hung Grinder processes on the machine at that time which an be seen in the log, so I wouldn't count that run as conclusive evidence, and (b) it seems to have been trying to load the AWT libraries so wasn't running in headless mode which could have caused additional problems.

My recommendation remains that if we're going to change this, we should be doing it for JDK11u/aarch64 ONLY.

andrew-m-leonard · 2022-07-01T13:41:28Z

We believe this is just a jdk11u problem, so updated PR to only remove from jdk11u.

andrew-m-leonard · 2022-07-01T13:41:48Z

run tests

eclipse-temurin-bot · 2022-07-01T13:42:03Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

andrew-m-leonard · 2022-07-01T13:48:01Z

run tests

eclipse-temurin-bot · 2022-07-01T13:48:14Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

Signed-off-by: Andrew Leonard <anleonar@redhat.com>

andrew-m-leonard · 2022-07-01T14:01:10Z

run tests

eclipse-temurin-bot · 2022-07-01T14:01:25Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

andrew-m-leonard · 2022-07-01T14:02:01Z

run tests

sxa

I'm +1 on this change after restricting to JDK11u/aarch64.
We have adoptium/temurin-build#2961 to cover the development of a fix, after which we can re-enable.

eclipse-temurin-bot · 2022-07-01T15:51:03Z

 PR TESTER RESULT 

❎ Some pipelines failed or the job was aborted! ❎
See the pipeline-build-check below for more information...

github-actions bot added jenkins-pipeline aarch testing labels Jun 29, 2022

sophia-guo approved these changes Jun 29, 2022

View reviewed changes

andrew-m-leonard force-pushed the alpine branch from 14f06d2 to 640554d Compare July 1, 2022 13:39

github-actions bot added testing and removed testing labels Jul 1, 2022

andrew-m-leonard changed the title ~~Remove aarch64Alpine from default pipeline config due to test hangs reliability~~ Remove aarch64Alpine from default jdk11u pipeline config due to test hangs reliability Jul 1, 2022

Remove aarch64AlpineLinux from default jdk11u pipeline configurations

02ccf78

Signed-off-by: Andrew Leonard <anleonar@redhat.com>

andrew-m-leonard force-pushed the alpine branch from 640554d to 02ccf78 Compare July 1, 2022 14:00

github-actions bot added testing and removed testing labels Jul 1, 2022

sxa approved these changes Jul 1, 2022

View reviewed changes

andrew-m-leonard merged commit 0d9963b into adoptium:master Jul 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove aarch64Alpine from default jdk11u pipeline config due to test hangs reliability #349

Remove aarch64Alpine from default jdk11u pipeline config due to test hangs reliability #349

andrew-m-leonard commented Jun 29, 2022 •

edited

Loading

karianna commented Jun 29, 2022

github-actions bot commented Jun 29, 2022

andrew-m-leonard commented Jun 29, 2022

sxa commented Jun 30, 2022

andrew-m-leonard commented Jun 30, 2022

sxa commented Jun 30, 2022

andrew-m-leonard commented Jun 30, 2022

andrew-m-leonard commented Jun 30, 2022

sxa commented Jun 30, 2022 •

edited

Loading

andrew-m-leonard commented Jun 30, 2022

sxa commented Jun 30, 2022 •

edited

Loading

andrew-m-leonard commented Jun 30, 2022

eclipse-temurin-bot commented Jun 30, 2022

sxa commented Jun 30, 2022

sophia-guo commented Jun 30, 2022

sxa commented Jul 1, 2022

andrew-m-leonard commented Jul 1, 2022

andrew-m-leonard commented Jul 1, 2022

eclipse-temurin-bot commented Jul 1, 2022

andrew-m-leonard commented Jul 1, 2022

eclipse-temurin-bot commented Jul 1, 2022

andrew-m-leonard commented Jul 1, 2022

eclipse-temurin-bot commented Jul 1, 2022

andrew-m-leonard commented Jul 1, 2022

sxa left a comment

eclipse-temurin-bot commented Jul 1, 2022

Remove aarch64Alpine from default jdk11u pipeline config due to test hangs reliability #349

Remove aarch64Alpine from default jdk11u pipeline config due to test hangs reliability #349

Conversation

andrew-m-leonard commented Jun 29, 2022 • edited Loading

karianna commented Jun 29, 2022

github-actions bot commented Jun 29, 2022

Thank you for creating a pull request!

Code Quality and Contributing Guidelines

Tests

andrew-m-leonard commented Jun 29, 2022

sxa commented Jun 30, 2022

andrew-m-leonard commented Jun 30, 2022

sxa commented Jun 30, 2022

andrew-m-leonard commented Jun 30, 2022

andrew-m-leonard commented Jun 30, 2022

sxa commented Jun 30, 2022 • edited Loading

andrew-m-leonard commented Jun 30, 2022

sxa commented Jun 30, 2022 • edited Loading

andrew-m-leonard commented Jun 30, 2022

eclipse-temurin-bot commented Jun 30, 2022

 PR TESTER RESULT 

sxa commented Jun 30, 2022

sophia-guo commented Jun 30, 2022

sxa commented Jul 1, 2022

andrew-m-leonard commented Jul 1, 2022

andrew-m-leonard commented Jul 1, 2022

eclipse-temurin-bot commented Jul 1, 2022

 PR TESTER RESULT 

andrew-m-leonard commented Jul 1, 2022

eclipse-temurin-bot commented Jul 1, 2022

 PR TESTER RESULT 

andrew-m-leonard commented Jul 1, 2022

eclipse-temurin-bot commented Jul 1, 2022

 PR TESTER RESULT 

andrew-m-leonard commented Jul 1, 2022

sxa left a comment

Choose a reason for hiding this comment

eclipse-temurin-bot commented Jul 1, 2022

 PR TESTER RESULT 

andrew-m-leonard commented Jun 29, 2022 •

edited

Loading

sxa commented Jun 30, 2022 •

edited

Loading

sxa commented Jun 30, 2022 •

edited

Loading