Discussion on how a test build is determined to be pass/fail

[AdamBrousseau]

When a make test command runs to completion without 'problems', it has a zero 0 exit code. The tests produce various artifacts where the results are stored. The test results summary make target analyzes all the results of the tests that were just run and prints to console a summary of the pass/fail including this line: TOTAL: XXX EXECUTED: XXX PASSED: XXX FAILED: XXX SKIPPED: XXX

When a Jenkins build runs make $TEST_TARGET by default, if the command completes with exit code 0, then as far as Jenkins is concerned, the "build" has passed. Various post build actions, some of which we currently use, will set a build to unstable. For example, Publish TestNG Results will examine all the testng reports and determine if there are test failures and set the build to unstable/failed based on configured thresholds. The problem is that a make test target can produce several different result artifacts (ex. testng, junit, tap) and it is not necessarily the most correct solution to have a post-build action for each result format to determine failures. Also, there is not necessarily a post-build test publisher step for every test result type the tests produce.

What we are guaranteed (currently) to have in the console each time is the test results summary. There is a post-build action we can use to search the console using a configured regex and set the build status appropriately. This would be a quick way to close current cracks in the Jenkins builds. However, I'm not sure if this is the correct solution either.

Two other options would be

• Have the original make command exit non-zero if there are failed tests
• Have a make command which can be run after a make test command which will exit non-zero if there were failures.

[AdamBrousseau]

Looks like we can't currently use text-finder-plugin in our pipeline jobs
Open Issue: https://issues.jenkins-ci.org/browse/JENKINS-34983
Open PR: jenkinsci/text-finder-plugin#6

[smlambert]

Please report whether the TAP (test anything protocol) plugin does the job or not.

[AdamBrousseau]

I can confirm that adding the TAP publish, to a build that previously only had testng and junit publish, will catch failure that were missed by testng and junit and set the build unstable. Since all our tests output, at minimum, TAP results (@smlambert please confirm) , I think using TAP as the default, with the addition of testng and junit (for prettiness) should suffice. I think the discussion could still be valuable since we are relying on 1+2 plugins to for test results.

I am curious to see what a PR build will report in the PR for pass/fail numbers when a TAP-only test fails.

[smlambert]

I confirm that we have TAP output for all of our tests, because we generate it for each individual test target (no matter what underlying framework was used). https://github.com/eclipse/openj9/blob/master/test/TestConfig/makefile#L281

In terms of reliance on plugins, the builds are already relying on plugins for supporting pipeline builds (despite some of those plugins being part of a modern Jenkins core set). The TAP and Junit plugins also help give a quick visual/graphical view that pure console view does not provide.

[smlambert]

But I agree, it might make sense for our final make target "resultsSummary" to write the summary, then assess whether there were failures reported and force a non-zero exit.

[AdamBrousseau]

Looks like pipeline support was just added for the text finder plugin via jenkinsci/text-finder-plugin#12.
Since we are removing the OpenJ9 test files and switching to Adopt (#1450 et al) maybe we should close this issue and open an issue at Adopt if we feel we want to switch to using this plugin for setting build status. Thoughts @smlambert @llxia ?

[smlambert]

I think the TAP plugin is sufficient for our purposes for now, but agree that we can close this issue.