Store executed tests on reports xml #235

imonteroperez · 2020-04-06T12:38:16Z

Provides the names of the executed tests under the PCT

[INFO] -------------------------------------------------------
[INFO] Solving test names
[INFO] -------------------------------------------------------
[INFO] Reading /tmp/pct/work/ant/target/surefire-reports
[INFO] Extracted 3 testnames from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks.AntWrapperTest.xml
[INFO] Extracted 1 testnames from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks._ant.AntTargetAnnotationTest.xml
[INFO] Extracted 3 testnames from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks._ant.AntTargetNoteTest.xml
[INFO] Extracted 15 testnames from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks.AntTest.xml
[INFO] Extracted 1 testnames from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks.AntJCasCCompatibilityTest.xml
[INFO] 
[INFO] Results:
[INFO] 
[INFO] - hudson.tasks.AntJCasCCompatibilityTest.roundTripTest
[INFO] - hudson.tasks.AntTest.customBuildFileTest
[INFO] - hudson.tasks.AntTest.emptyParameterTest
[INFO] - hudson.tasks.AntTest.invokeCustomTargetTest
[INFO] - hudson.tasks.AntTest.invokeDefaultTargetTest
[INFO] - hudson.tasks.AntTest.jenkinsEnvVarsFromBuildScriptTest
[INFO] - hudson.tasks.AntTest.optionsInTargetFieldTest
[INFO] - hudson.tasks.AntTest.propertyReplacedByEmptyBuildParameter
[INFO] - hudson.tasks.AntTest.propertyReplacedByVariable
[INFO] - hudson.tasks.AntTest.testConfigRoundtrip
[INFO] - hudson.tasks.AntTest.testEscapeXmlInParameters
[INFO] - hudson.tasks.AntTest.testGlobalConfigAjax
[INFO] - hudson.tasks.AntTest.testParameterExpansion
[INFO] - hudson.tasks.AntTest.testParameterExpansionByShell
[INFO] - hudson.tasks.AntTest.testSensitiveParameters
[INFO] - hudson.tasks.AntTest.unexistingCustomBuildFileTest
[INFO] - hudson.tasks.AntWrapperTest.configRoundTrip
[INFO] - hudson.tasks.AntWrapperTest.durability
[INFO] - hudson.tasks.AntWrapperTest.smokes
[INFO] - hudson.tasks._ant.AntTargetAnnotationTest.test1
[INFO] - hudson.tasks._ant.AntTargetNoteTest.testAnnotateTarget
[INFO] - hudson.tasks._ant.AntTargetNoteTest.testAnnotateTargetContainingColon
[INFO] - hudson.tasks._ant.AntTargetNoteTest.testDisabled

Stores the info inside the xml reports

<compatResult>
   <coreCoordinates>
      <groupId>org.jenkins-ci.plugins</groupId>
      <artifactId>plugin</artifactId>
      <version>2.222.1</version>
   </coreCoordinates>
   <status>SUCCESS</status>
   <compatTestExecutedOn>2020-04-06 10:28:57.703 UTC</compatTestExecutedOn>
   <warningMessages />
   <executedTests>
      <string>hudson.tasks.AntTest.optionsInTargetFieldTest</string>
      <string>hudson.tasks.AntTest.invokeDefaultTargetTest</string>
      <string>hudson.tasks.AntWrapperTest.smokes</string>
      <string>hudson.tasks.AntTest.testEscapeXmlInParameters</string>
      <string>hudson.tasks.AntTest.customBuildFileTest</string>
      <string>hudson.tasks.AntTest.jenkinsEnvVarsFromBuildScriptTest</string>
      <string>hudson.tasks.AntTest.testConfigRoundtrip</string>
      <string>hudson.tasks._ant.AntTargetNoteTest.testAnnotateTarget</string>
      <string>hudson.tasks._ant.AntTargetNoteTest.testDisabled</string>
      <string>hudson.tasks._ant.AntTargetNoteTest.testAnnotateTargetContainingColon</string>
      <string>hudson.tasks.AntTest.propertyReplacedByEmptyBuildParameter</string>
      <string>hudson.tasks.AntTest.testParameterExpansion</string>
      <string>hudson.tasks.AntTest.testParameterExpansionByShell</string>
      <string>hudson.tasks.AntWrapperTest.configRoundTrip</string>
      <string>hudson.tasks.AntTest.testGlobalConfigAjax</string>
      <string>hudson.tasks.AntJCasCCompatibilityTest.roundTripTest</string>
      <string>hudson.tasks.AntTest.unexistingCustomBuildFileTest</string>
      <string>hudson.tasks.AntTest.propertyReplacedByVariable</string>
      <string>hudson.tasks.AntTest.testSensitiveParameters</string>
      <string>hudson.tasks._ant.AntTargetAnnotationTest.test1</string>
      <string>hudson.tasks.AntTest.invokeCustomTargetTest</string>
      <string>hudson.tasks.AntWrapperTest.durability</string>
      <string>hudson.tasks.AntTest.emptyParameterTest</string>
   </executedTests>
   <buildLogPath>logs/ant/v1.11_against_org.jenkins-ci.plugins_plugin_2.222.1.log</buildLogPath>
</compatResult>

oleg-nenashev

Thanks, it would be a great improvement! At the same time it may bloat reports and files for plugins which have thousands of tests and configuration combinations. It would be great to make this data an opt-in feature.

E.g.:

Store/log failed tests by defult
Store/log passed tests only if there is a flag set

imonteroperez · 2020-04-07T10:03:19Z

Thanks, it would be a great improvement! At the same time it may bloat reports and files for plugins which have thousands of tests and configuration combinations. It would be great to make this data an opt-in feature.

E.g.:

Store/log failed tests by defult

Store/log passed tests only if there is a flag set

Love that idea, I will update providing support on classifying tests, thanks!

imonteroperez · 2020-04-07T13:33:11Z

After applying @oleg-nenashev proposal here: 7ee332a, here is an example with some ant unit tests modified with assertEquals(true,false) to force some failures:

[INFO] -------------------------------------------------------
[INFO] Solving test names
[INFO] -------------------------------------------------------
[INFO] Reading /tmp/pct/work/ant/target/surefire-reports
[INFO] Extracted 3 testnames from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks.AntWrapperTest.xml
[WARNING] Extracted: 1, Expected: 5 from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks._ant.AntTargetAnnotationTest.xml
[INFO] Extracted 3 testnames from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks._ant.AntTargetNoteTest.xml
[WARNING] Extracted: 15, Expected: 19 from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks.AntTest.xml
[INFO] Extracted 1 testnames from /tmp/pct/work/ant/target/surefire-reports/TEST-hudson.tasks.AntJCasCCompatibilityTest.xml
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Executed: 21
[INFO] - hudson.tasks.AntJCasCCompatibilityTest.roundTripTest
[INFO] - hudson.tasks.AntTest.customBuildFileTest
[INFO] - hudson.tasks.AntTest.emptyParameterTest
[INFO] - hudson.tasks.AntTest.invokeCustomTargetTest
[INFO] - hudson.tasks.AntTest.invokeDefaultTargetTest
[INFO] - hudson.tasks.AntTest.jenkinsEnvVarsFromBuildScriptTest
[INFO] - hudson.tasks.AntTest.optionsInTargetFieldTest
[INFO] - hudson.tasks.AntTest.propertyReplacedByEmptyBuildParameter
[INFO] - hudson.tasks.AntTest.propertyReplacedByVariable
[INFO] - hudson.tasks.AntTest.testEscapeXmlInParameters
[INFO] - hudson.tasks.AntTest.testGlobalConfigAjax
[INFO] - hudson.tasks.AntTest.testParameterExpansion
[INFO] - hudson.tasks.AntTest.testParameterExpansionByShell
[INFO] - hudson.tasks.AntTest.testSensitiveParameters
[INFO] - hudson.tasks.AntTest.unexistingCustomBuildFileTest
[INFO] - hudson.tasks.AntWrapperTest.configRoundTrip
[INFO] - hudson.tasks.AntWrapperTest.durability
[INFO] - hudson.tasks.AntWrapperTest.smokes
[INFO] - hudson.tasks._ant.AntTargetNoteTest.testAnnotateTarget
[INFO] - hudson.tasks._ant.AntTargetNoteTest.testAnnotateTargetContainingColon
[INFO] - hudson.tasks._ant.AntTargetNoteTest.testDisabled
[INFO] 
[INFO] Failed: 2
[INFO] - hudson.tasks.AntTest.testConfigRoundtrip
[INFO] - hudson.tasks._ant.AntTargetAnnotationTest.test1

Generated report by default:

<testDetails class="sorted-set">
       <string>hudson.tasks.AntTest.testConfigRoundtrip</string>
       <string>hudson.tasks._ant.AntTargetAnnotationTest.test1</string>
</testDetails>

If user setup -storeAll option on running the PCT report will provide all the tests inside testDetails entry.

…s is needed on ci.jenkins.io

raul-arabaolaza · 2020-04-20T09:00:28Z

Given that the changes requested by @oleg-nenashev have been implemented and the builder is green I am going to merge

imonteroperez added 8 commits April 6, 2020 13:56

Store executed tests on reports xml

b538b9e

Added unit tests

3f8eee7

Fixed unit test

b14c24b

Using accurev

c5141d0

Using accurev

2b1d76d

Fix issue on document location

75d452e

Support on test failures and some unit tests

561744e

Testing with accurev

ab22fec

oleg-nenashev requested changes Apr 7, 2020

View reviewed changes

imonteroperez added 2 commits April 7, 2020 11:53

Enabled incrementals

5030b8a

Fixed unit test

85563d7

imonteroperez added 3 commits April 7, 2020 13:26

Support for clasiffy test failures

7ee332a

Test with workflow-api

611f91d

Refactor test

a28606f

imonteroperez added 14 commits April 7, 2020 16:06

Restore ant

56a84cd

Restore ant

94b76a8

Debug

625dd77

Fixed tests

af17ee2

Using external maven settings

b2025a5

Using build-trigger-badge

a1c4c30

Unit tests working properly locally. Need help to review what setting…

8ee8cfb

…s is needed on ci.jenkins.io

Using specific location folder

62a6ffc

Retrieve settingsXml file

48324dd

Debug jenkinsfile

4c67b26

Debug

c89a72a

Restored without debug messages

8b295f4

Restored Jenkinsfile

25d386a

Force debug

d229b17

imonteroperez added 14 commits April 13, 2020 17:04

Force rebuild on ci

126f059

Debug exception on runner in Windows

fb128c9

Setup external maven runner

ebcc13a

Setup maven config

fefb8e4

Setup for linux

19b006f

Setup for windows

4e10923

Restore linux

72a96f4

Testing windows path

1af8ec2

Setup windows location

0b8ee0c

Fix windows

bfd1003

Removed unused import

c0ef658

Merge remote-tracking branch 'upstream/master' into pct-store-test-names

e546e61

Force reload

f5ff126

Force ci.jenkins.io

9ee1583

imonteroperez requested a review from oleg-nenashev April 15, 2020 10:09

raul-arabaolaza approved these changes Apr 15, 2020

View reviewed changes

bmunozm approved these changes Apr 16, 2020

View reviewed changes

raul-arabaolaza merged commit 4322009 into jenkinsci:master Apr 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store executed tests on reports xml #235

Store executed tests on reports xml #235

imonteroperez commented Apr 6, 2020

oleg-nenashev left a comment

imonteroperez commented Apr 7, 2020

imonteroperez commented Apr 7, 2020

raul-arabaolaza commented Apr 20, 2020

Store executed tests on reports xml #235

Store executed tests on reports xml #235

Conversation

imonteroperez commented Apr 6, 2020

oleg-nenashev left a comment

Choose a reason for hiding this comment

imonteroperez commented Apr 7, 2020

imonteroperez commented Apr 7, 2020

raul-arabaolaza commented Apr 20, 2020