Bugfix on retrieving plugin test types #271

imonteroperez · 2021-02-08T10:30:51Z

This PR provides a bugfix on generating the report of failed tests based on its type

Context

On Support for evaluating additional test goals #258 support for evaluating additional test goals was introduced. It provides to the PCT the capabilities to evaluate not only surefire based tests, also others like failsafe

Problem statement

Once used this new support, detected that PCT plugin report was not properly showing all the test failures
@bmunozm did a preliminary analysis and detected this via bisect that reason was about mentioned PR
Example of invalid result with google-compute-engine plugin:

[...]
executed classname tests: [com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest, com.google.jenkins.plugins.computeengine.AcceleratorConfigurationTest, com.google.jenkins.plugins.computeengine.GoogleKeyPairTest, com.google.jenkins.plugins.computeengine.ConfigAsCodeTest, com.google.jenkins.plugins.computeengine.InstanceConfigurationTest, com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest, com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest, com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest]
[INFO] -------------------------------------------------------
[INFO] Solving test names
[INFO] -------------------------------------------------------
[WARNING] No test reports found!
[...]

Example of expected result for google-compute-engine plugin:

[...]
executed classname tests: [com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest, com.google.jenkins.plugins.computeengine.AcceleratorConfigurationTest, com.google.jenkins.plugins.computeengine.GoogleKeyPairTest, com.google.jenkins.plugins.computeengine.ConfigAsCodeTest, com.google.jenkins.plugins.computeengine.InstanceConfigurationTest, com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest, com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest, com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest]
[INFO] -------------------------------------------------------
[INFO] Solving test names
[INFO] -------------------------------------------------------
[INFO] Reading /pct/tmp/work/google-compute-engine/target/surefire-reports
[INFO] Extracted 4 testnames from /pct/tmp/work/google-compute-engine/target/surefire-reports/TEST-com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.xml
[INFO] Extracted 2 testnames from /pct/tmp/work/google-compute-engine/target/surefire-reports/TEST-com.google.jenkins.plugins.computeengine.AcceleratorConfigurationTest.xml
[INFO] Extracted 1 testnames from /pct/tmp/work/google-compute-engine/target/surefire-reports/TEST-com.google.jenkins.plugins.computeengine.GoogleKeyPairTest.xml
[INFO] Extracted 2 testnames from /pct/tmp/work/google-compute-engine/target/surefire-reports/TEST-com.google.jenkins.plugins.computeengine.ConfigAsCodeTest.xml
[INFO] Extracted 6 testnames from /pct/tmp/work/google-compute-engine/target/surefire-reports/TEST-com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.xml
[INFO] Extracted 9 testnames from /pct/tmp/work/google-compute-engine/target/surefire-reports/TEST-com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.xml
[INFO] Extracted 5 testnames from /pct/tmp/work/google-compute-engine/target/surefire-reports/TEST-com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.xml
[INFO] Extracted 6 testnames from /pct/tmp/work/google-compute-engine/target/surefire-reports/TEST-com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.xml
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Executed: 34
[INFO] - com.google.jenkins.plugins.computeengine.AcceleratorConfigurationTest.clientCalls
[INFO] - com.google.jenkins.plugins.computeengine.AcceleratorConfigurationTest.construction
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testConstructor
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoCheckNetworkItemsEmpty
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoCheckNetworkItemsValid
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoCheckSubnetworkItemsEmpty
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoCheckSubnetworkItemsValid
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoFillNetworkItems
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoFillSubnetworkItemsEmptyRegion
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoFillSubnetworkItemsNoSubnetworks
[INFO] - com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoFillSubnetworkItemsValid
[INFO] - com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldCleanLostInstance
[INFO] - com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldNotCleanAnyInstance
[INFO] - com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldNotCleanStoppingInstance
[INFO] - com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldRegisterCleanNodeWorker
[INFO] - com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldRunWithoutClouds
[INFO] - com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.construction
[INFO] - com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.descriptorFillCredentials
[INFO] - com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.descriptorProjectValidation
[INFO] - com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.getConfigurationsByLabelMulti
[INFO] - com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.getConfigurationsByLabelSimple
[INFO] - com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.instanceIdWasGenerated
[INFO] - com.google.jenkins.plugins.computeengine.ConfigAsCodeTest.shouldCreateCloudInstanceFromCode
[INFO] - com.google.jenkins.plugins.computeengine.ConfigAsCodeTest.shouldCreateGCEClientFromCode
[INFO] - com.google.jenkins.plugins.computeengine.GoogleKeyPairTest.KeyPairGeneration
[INFO] - com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.descriptorBootDiskSizeValidation
[INFO] - com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.testClient
[INFO] - com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.testConfigRoundtrip
[INFO] - com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.testInstanceMetadata
[INFO] - com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.testInstanceModel
[INFO] - com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.construction
[INFO] - com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.descriptorProjectId
[INFO] - com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.descriptorRegion
[INFO] - com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.descriptorSubnetwork
[INFO] 
[INFO] Failed: 1
[INFO] - com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.unnecessary Mockito stubbings
[...]

Solution and implementation details

Detected wrongly null injection of types values to the solver
Injected types as part of the userProperties map inside the config provided to the runner
Results obtained as expected
Example of generated report:

<entry>
  <pluginInfos>
    <pluginName>google-compute-engine</pluginName>
    <pluginVersion>4.3.3</pluginVersion>
    <pluginUrl>jar:file:/pct/tmp/jenkins.war!/WEB-INF/plugins/google-compute-engine.hpi</pluginUrl>
  </pluginInfos>
  <list>
    <compatResult>
      [...]
      <status>TEST_FAILURES</status>
      [...]
      <testDetails class="sorted-set">
        <string>com.google.jenkins.plugins.computeengine.AcceleratorConfigurationTest.clientCalls</string>
        <string>com.google.jenkins.plugins.computeengine.AcceleratorConfigurationTest.construction</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testConstructor</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoCheckNetworkItemsEmpty</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoCheckNetworkItemsValid</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoCheckSubnetworkItemsEmpty</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoCheckSubnetworkItemsValid</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoFillNetworkItems</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoFillSubnetworkItemsEmptyRegion</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoFillSubnetworkItemsNoSubnetworks</string>
        <string>com.google.jenkins.plugins.computeengine.AutofilledNetworkConfigurationTest.testDoFillSubnetworkItemsValid</string>
        <string>com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldCleanLostInstance</string>
        <string>com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldNotCleanAnyInstance</string>
        <string>com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldNotCleanStoppingInstance</string>
        <string>com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldRegisterCleanNodeWorker</string>
        <string>com.google.jenkins.plugins.computeengine.CleanLostNodesWorkTest.shouldRunWithoutClouds</string>
        <string>com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.construction</string>
        <string>com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.descriptorFillCredentials</string>
        <string>com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.descriptorProjectValidation</string>
        <string>com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.getConfigurationsByLabelMulti</string>
        <string>com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.getConfigurationsByLabelSimple</string>
        <string>com.google.jenkins.plugins.computeengine.ComputeEngineCloudTest.instanceIdWasGenerated</string>
        <string>com.google.jenkins.plugins.computeengine.ConfigAsCodeTest.shouldCreateCloudInstanceFromCode</string>
        <string>com.google.jenkins.plugins.computeengine.ConfigAsCodeTest.shouldCreateGCEClientFromCode</string>
        <string>com.google.jenkins.plugins.computeengine.GoogleKeyPairTest.KeyPairGeneration</string>
        <string>com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.descriptorBootDiskSizeValidation</string>
        <string>com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.testClient</string>
        <string>com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.testConfigRoundtrip</string>
        <string>com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.testInstanceMetadata</string>
        <string>com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.testInstanceModel</string>
        <string>com.google.jenkins.plugins.computeengine.InstanceConfigurationTest.unnecessary Mockito stubbings</string>
        <string>com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.construction</string>
        <string>com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.descriptorProjectId</string>
        <string>com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.descriptorRegion</string>
        <string>com.google.jenkins.plugins.computeengine.SharedVpcNetworkConfigurationTest.descriptorSubnetwork</string>
      </testDetails>
      [...]
    </compatResult>
  </list>
</entry>

@raul-arabaolaza @bmunozm @batmat

batmat · 2021-02-08T11:30:47Z

IIUC, if we missed this regression, I would love to see a test reproducing this issue.
Can we add an IT inside https://github.com/jenkinsci/plugin-compat-tester/blob/master/Jenkinsfile#L55 so we cover this case?
Thanks

oleg-nenashev · 2021-02-08T11:40:37Z

plugins-compat-tester/src/main/java/org/jenkins/tools/test/maven/ExternalMavenRunner.java

@@ -113,4 +113,17 @@ public void run(Config config, File baseDirectory, File buildLogFile, String...
        }
    }

+    private Set<String> getTypes(Config config) {
+        if (config == null || config.userProperties == null || !config.userProperties.containsKey("types")) {
+            return null;


Default to surefire?

Updated to use surefire as default here: 9f198e6

imonteroperez · 2021-02-08T14:38:55Z

IIUC, if we missed this regression, I would love to see a test reproducing this issue.
Can we add an IT inside https://github.com/jenkinsci/plugin-compat-tester/blob/master/Jenkinsfile#L55 so we cover this case?
Thanks

@batmat added an IT on 28ec40c but seems Jenkinsfile updates are not evaluated in the PR builder right? https://ci.jenkins.io/blue/organizations/jenkins/jenkinsci-libraries%2Fplugin-compat-tester/detail/PR-271/5/pipeline/

batmat · 2021-02-08T20:00:30Z

@imonteroperez I've refiled your IT as #272

TBH, for next times, I think you should be made an official maintainer, given that's not your first PR and you've got a special interest to make this repository work nicely :):

https://github.com/jenkinsci/plugin-compat-tester/pulls?q=is%3Apr+author%3Aimonteroperez

And same goes probably for @bmunozm BTW, given https://github.com/jenkinsci/plugin-compat-tester/pulls?q=is%3Apr+author%3Abmunozm

@raul-arabaolaza @oleg-nenashev @jglick as currently declared maintainers, would some of you approve this? Thanks

jglick · 2021-02-08T21:23:26Z

as currently declared maintainers

I contribute on occasion but would not consider myself a maintainer.

imonteroperez · 2021-02-10T09:13:27Z

@imonteroperez I've refiled your IT as #272

TBH, for next times, I think you should be made an official maintainer, given that's not your first PR and you've got a special interest to make this repository work nicely :):

https://github.com/jenkinsci/plugin-compat-tester/pulls?q=is%3Apr+author%3Aimonteroperez

And same goes probably for @bmunozm BTW, given https://github.com/jenkinsci/plugin-compat-tester/pulls?q=is%3Apr+author%3Abmunozm

@raul-arabaolaza @oleg-nenashev @jglick as currently declared maintainers, would some of you approve this? Thanks

IT worked as expected now: https://ci.jenkins.io/blue/organizations/jenkins/jenkinsci-libraries%2Fplugin-compat-tester/detail/PR-272/4/pipeline/171/

raul-arabaolaza · 2021-02-10T10:21:38Z

@imonteroperez I've refiled your IT as #272

TBH, for next times, I think you should be made an official maintainer, given that's not your first PR and you've got a special interest to make this repository work nicely :):

https://github.com/jenkinsci/plugin-compat-tester/pulls?q=is%3Apr+author%3Aimonteroperez

And same goes probably for @bmunozm BTW, given https://github.com/jenkinsci/plugin-compat-tester/pulls?q=is%3Apr+author%3Abmunozm

@raul-arabaolaza @oleg-nenashev @jglick as currently declared maintainers, would some of you approve this? Thanks

I do agree, both Bea and Ilde are as much as contributors as I am myself.

batmat · 2021-02-10T10:22:18Z

OK, I'll add them then. Thanks

imonteroperez added 2 commits February 8, 2021 11:16

Bugfix on retrieving plugin test types

15064d4

Remove extra blank spaces

9fc13af

oleg-nenashev approved these changes Feb 8, 2021

View reviewed changes

imonteroperez added 4 commits February 8, 2021 14:14

Use surefire as default value

9f198e6

Remove unused import Strings

bc9a8cb

Added IT

28ec40c

Fix stage name

676119b

batmat mentioned this pull request Feb 8, 2021

Integration Test for retrieving plugin test types #272

Closed

bmunozm approved these changes Feb 8, 2021

View reviewed changes

imonteroperez added 2 commits February 9, 2021 11:20

Fix sha

753a449

Fail on error on IT

9736b89

raul-arabaolaza merged commit df60aee into jenkinsci:master Feb 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix on retrieving plugin test types #271

Bugfix on retrieving plugin test types #271

imonteroperez commented Feb 8, 2021 •

edited

Loading

batmat commented Feb 8, 2021

oleg-nenashev Feb 8, 2021

imonteroperez Feb 8, 2021

imonteroperez commented Feb 8, 2021

batmat commented Feb 8, 2021 •

edited

Loading

jglick commented Feb 8, 2021

imonteroperez commented Feb 10, 2021

raul-arabaolaza commented Feb 10, 2021

batmat commented Feb 10, 2021

Bugfix on retrieving plugin test types #271

Bugfix on retrieving plugin test types #271

Conversation

imonteroperez commented Feb 8, 2021 • edited Loading

Context

Problem statement

Solution and implementation details

batmat commented Feb 8, 2021

oleg-nenashev Feb 8, 2021

Choose a reason for hiding this comment

imonteroperez Feb 8, 2021

Choose a reason for hiding this comment

imonteroperez commented Feb 8, 2021

batmat commented Feb 8, 2021 • edited Loading

jglick commented Feb 8, 2021

imonteroperez commented Feb 10, 2021

raul-arabaolaza commented Feb 10, 2021

batmat commented Feb 10, 2021

imonteroperez commented Feb 8, 2021 •

edited

Loading

batmat commented Feb 8, 2021 •

edited

Loading