Build passes even after PCT fails to run on a plugin

[jglick]

https://github.com/jenkinsci/bom/pull/893/checks?check_run_id=5266351579 did not run any tests. https://ci.jenkins.io/job/Tools/job/bom/job/PR-893/1/execution/node/5973/log/ “passed” yet

…
 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test (default-cli) on project script-security: Execution default-cli of goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M4:test failed: java.lang.NoClassDefFoundError: com/google/common/io/NullOutputStream: com.google.common.io.NullOutputStream -> [Help 1]
 [ERROR] 
 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
 [ERROR] Re-run Maven using the -X switch to enable full debug logging.
 [ERROR] 
 [ERROR] For more information about the errors and possible solutions, please read the following articles:
 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
 succeeded artifactIds: [maven-hpi-plugin, maven-hpi-plugin]
 executed classname tests: []
 [INFO] -------------------------------------------------------
 [INFO] Solving test names
 [INFO] -------------------------------------------------------
 [WARNING] No test reports found!
 org.jenkins.tools.test.exception.PomExecutionException: [mvn, --show-version, --batch-mode, -ntp, --settings=/home/jenkins/workspace/Tools_bom_PR-893@tmp/settings-azure.xml, --define=failIfNoTests=false, --define=forkCount=.75C, --define=jth.jenkins-war.path=/home/jenkins/workspace/Tools_bom_PR-893/megawar.war, --define=surefire.excludesFile=/home/jenkins/workspace/Tools_bom_PR-893/excludes.txt, --define=surefire.rerunFailingTestsCount=4, --define=types=failsafe,surefire, --define=forkCount=.75C, hpi:resolve-test-dependencies, hpi:test-hpl, surefire:test, -Djenkins.version=2.335, -Denforcer.skip=true, failsafe:integration-test] failed in /home/jenkins/workspace/Tools_bom_PR-893/pct-work/script-security
 	at org.jenkins.tools.test.maven.ExternalMavenRunner.run(ExternalMavenRunner.java:105)
 	at org.jenkins.tools.test.PluginCompatTester.testPluginAgainst(PluginCompatTester.java:569)
 	at org.jenkins.tools.test.PluginCompatTester.testPlugins(PluginCompatTester.java:312)
 	at org.jenkins.tools.test.PluginCompatTesterCli.main(PluginCompatTesterCli.java:178)
 Plugin slack not in included plugins => test skipped !
…
 + grep -q -F -e '<status>INTERNAL_ERROR</status>' pct-report.xml
 + grep -q -F -e '<status>TEST_FAILURES</status>' pct-report.xml
 + rm -fv 'pct-work/*/target/surefire-reports/TEST-InjectedTest.xml' 'pct-work/*/*/target/surefire-reports/TEST-InjectedTest.xml'

[jglick]

Reproducible as of 129e836 via

PLUGINS=script-security LINE=weekly bash local-test.sh

(note not setting TEST).

I suppose this is a bug in PCT.

bom/pct.sh

Lines 38 to 49 in 129e836

	echo PCT marked failed, checking to see if that is due to a failure to run tests at all
	for t in pct-work/*/{,*/}target
	do
	if [ -f $t/test-classes/InjectedTest.class -a \! -f $t/surefire-reports/TEST-InjectedTest.xml ]
	then
	mkdir -p $t/surefire-reports
	cat > $t/surefire-reports/TEST-pct.xml <<'EOF'
	<testsuite name="pct">
	<testcase classname="pct" name="overall">
	<error message="some sort of PCT problem; look at logs"/>
	</testcase>
	</testsuite>

is not getting triggered because pct-report.xml does not claim that PCT failed at all:

          <status>SUCCESS</status>
          <errorMessage>Message : [mvn, --show-version, --batch-mode, -ntp, --define=failIfNoTests=false, --define=forkCount=.75C, --define=jth.jenkins-war.path=…bom/target/local-test/megawar.war, --define=surefire.excludesFile=…bom/target/local-test/excludes.txt, --define=types=failsafe,surefire, --define=forkCount=.75C, hpi:resolve-test-dependencies, hpi:test-hpl, surefire:test, failsafe:integration-test, -Djenkins.version=2.335, -Denforcer.skip=true] failed in …bom/target/local-test/pct-work/script-security 
 
Executed plugins : [maven-hpi-plugin, maven-hpi-plugin, maven-compiler-plugin] 
 
Stacktraces :
</errorMessage>

Cannot tell what the root error is in Surefire since PCT is not passing -e to get a stack trace.

[jglick]

(jenkinsci/script-security-plugin#387 is supposed to fix the plugin incompatibility, but the point here is that PCT in bom/Jenkinsfile should have caught this earlier.)

[jglick]

jenkinsci/plugin-compat-tester#347 makes the problem clearer from the build log. Maybe also fixes this problem? Seems to produce

<status>INTERNAL_ERROR</status>

though I am not sure how --errors could accomplish that. Just the presence of the stack trace? Having difficulty following the logic in ExecutedTestNamesDetails.

[jglick]

Have some proposed fixes in PCT. If those are both accepted, need to merge trunk into jenkinsci/plugin-compat-tester#345 and check that update here to see if it correctly catches errors such as that from script-security in weekly.

[basil]

Why is this filed under jenkinsci/bom? If this is a PCT issue, it should be filed in that component.

[jglick]

Possibly, pending evaluation that the fault does not lie in

bom/pct.sh

Lines 29 to 47 in 3e42bee

	if grep -q -F -e '<status>INTERNAL_ERROR</status>' pct-report.xml; then
	echo PCT failed
	cat pct-report.xml
	exit 1
	elif grep -q -F -e '<status>TEST_FAILURES</status>' pct-report.xml; then
	echo PCT marked failed, checking to see if that is due to a failure to run tests at all
	for t in pct-work/*/{,*/}target; do
	if [[ -f "${t}/test-classes/InjectedTest.class" ]] && [[ ! -f "${t}/surefire-reports/TEST-InjectedTest.xml" ]]; then
	mkdir -p "${t}/surefire-reports"
	cat >"${t}/surefire-reports/TEST-pct.xml" <<-'EOF'
	<testsuite name="pct">
	<testcase classname="pct" name="overall">
	<error message="some sort of PCT problem; look at logs"/>
	</testcase>
	</testsuite>
	EOF
	fi
	done
	fi

[basil]

At minimum

bom/pct.sh

Line 47 in 4989785

if [[ -f "${t}/test-classes/InjectedTest.class" ]] && [[ ! -f "${t}/surefire-reports/TEST-InjectedTest.xml" ]]; then

needs to account for https://github.com/jenkinsci/analysis-pom-plugin/blob/a888caccf30603b8dcceb710ead8530baf55b76e/pom.xml#L244-L280 by checking for failsafe-reports/TEST-InjectedTest.xml.

[jglick]

PLUGINS=mailer TEST=InjectedTest bash local-test.sh

seems to leave behind no indication of a failure after #1160 despite output clearly showing that PCT generated an invalid POM that could not be compiled, much less tested:

[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for org.jenkins-ci.modules:instance-identity:jar is missing. @ org.jenkins-ci.plugins:mailer:${changelist}, …/target/local-test/pct-work/mailer/pom.xml, line 61, column 17
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project org.jenkins-ci.plugins:mailer:999999-SNAPSHOT (…/target/local-test/pct-work/mailer/pom.xml) has 1 error
[ERROR]     'dependencies.dependency.version' for org.jenkins-ci.modules:instance-identity:jar is missing. @ org.jenkins-ci.plugins:mailer:${changelist}, …/target/local-test/pct-work/mailer/pom.xml, line 61, column 17
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
succeeded artifactIds: []
executed classname tests: []
[INFO] -------------------------------------------------------
[INFO] Solving test names
[INFO] -------------------------------------------------------
[WARNING] No test reports found!
org.jenkins.tools.test.exception.PomExecutionException: [mvn, --show-version, --batch-mode, -ntp, --define=failIfNoTests=false, --define=forkCount=.75C, --define=jth.jenkins-war.path=…/target/local-test/megawar.war, --define=surefire.excludesFile=…/target/local-test/excludes.txt, --define=test=InjectedTest, --define=types=failsafe,surefire, --define=forkCount=.75C, hpi:resolve-test-dependencies, hpi:test-hpl, surefire:test, failsafe:integration-test, -Djenkins.version=2.359, -Denforcer.skip=true] failed in …/target/local-test/pct-work/mailer
	at org.jenkins.tools.test.maven.ExternalMavenRunner.run(ExternalMavenRunner.java:105)
	at org.jenkins.tools.test.PluginCompatTester.testPluginAgainst(PluginCompatTester.java:569)
	at org.jenkins.tools.test.PluginCompatTester.testPlugins(PluginCompatTester.java:312)
	at org.jenkins.tools.test.PluginCompatTesterCli.main(PluginCompatTesterCli.java:178)

PCT seems to have been at fault in this case:

<status>SUCCESS</status>
<compatTestExecutedOn>2022-07-19 12:56:20.64 UTC</compatTestExecutedOn>
<errorMessage>Message : [mvn, --show-version, --batch-mode, -ntp, --define=failIfNoTests=false, --define=forkCount=.75C, --define=jth.jenkins-war.path=…/target/local-test/megawar.war, --define=surefire.excludesFile=…/target/local-test/excludes.txt, --define=test=InjectedTest, --define=types=failsafe,surefire, --define=forkCount=.75C, hpi:resolve-test-dependencies, hpi:test-hpl, surefire:test, failsafe:integration-test, -Djenkins.version=2.359, -Denforcer.skip=true] failed in …/target/local-test/pct-work/mailer 
 
Executed plugins : [maven-compiler-plugin] 
 
Stacktraces :
</errorMessage>
<warningMessages/>

To do: compare behavior under #1059.

[jglick]

Same error for sshd. git-server does not have such a problem, apparently because its POM is old enough to still specify an explicit

<version>2.2</version>

[jglick]

Was unable to confirm that this was fixed, insofar as neither of the previously reported examples produce an error at all now. Good enough.

[jetersen]

@jglick see #1337 ? 🤔

[basil]

I did not intentionally close this - apparently GitHub aggressively closed it based on me writing this in #1336:

Does not fix #895, but brings us closer to a solution

[basil]

This time, closing the issue was the expected behavior. :-)