use a shallow clone as we do not need history here (do not use anymore maven-scm but use more simple and faster direct git cli) #382

olamy · 2022-10-28T04:23:42Z

Signed-off-by: Olivier Lamy olamy@apache.org

pom.xml

plugins-compat-tester/pom.xml

jglick

Is the motivation to speed up PCT runs?

Probably OK for now. I am hesitant because this would cause -Dset.changelist to not work on JEP-229 plugins. Currently we do not pass that, but it would be one way to solve a problem with plugins in multimodule repositories. Another way, possibly better but also more intrusive, would be rearrange how plugins are collected and the tests are orchestrated to get rid of AbstractMultiParentHook and always run Maven from the repository root (presuming that is the aggregator POM!). I think not discussed explicitly but related to #318 (comment)

plugins-compat-tester/pom.xml

plugins-compat-tester/src/main/java/org/jenkins/tools/test/hook/AbstractMultiParentHook.java

olamy · 2022-10-29T01:05:55Z

Is the motivation to speed up PCT runs?

yes and it does a bit.
Maybe as a cli option?
a more significant improvement could be to download the sources only (maybe from gh release as the -sources artifacts from maven doesn’t have the test sources it's useless but this would have been great to use repository manager cache and local maven repository cache as well) and cache them locally in something configurable such ~/.uur.src.cache/ as those sources tarball are supposed to be immutable.
But yup still some issues with version of the pom which will not the same as the tag (can be solved maybe using -Dchangelist=
the problem I see is to detect do we have to pass this argument or not.

Probably OK for now. I am hesitant because this would cause -Dset.changelist to not work on JEP-229 plugins. Currently we do not pass that, but it would be one way to solve a problem with plugins in multimodule repositories. Another way, possibly better but also more intrusive, would be rearrange how plugins are collected and the tests are orchestrated to get rid of AbstractMultiParentHook and always run Maven from the repository root (presuming that is the aggregator POM!). I think not discussed explicitly but related to #318 (comment)

olamy · 2022-10-29T03:14:38Z

@jglick it looks I forgot the same change for mono module repo. so done

olamy · 2022-10-29T23:43:56Z

back to draft. as not sure this can even work with mono module repo because of the need to have a full git changelog history
to be able to get a version number to build a project. as per default the tagged source code (e.g the pom) could have a version number 999999-SNAPSHOT
and element could be wrong in some deployed pom as it contains the commit and not the git tag so pct currently use it as a tag

curl -s https://repo.jenkins-ci.org/artifactory/public/org/jenkins-ci/plugins/mailer/438.v02c7f0a_12fa_4/mailer-438.v02c7f0a_12fa_4.pom | grep tag 
    <tag>02c7f0a12fa48e72973a9da4741ecdc59b2d8fe5</tag>

need more changes to accommodate this feature.

jglick · 2022-10-31T12:28:22Z

the tagged source code (e.g the pom) could have a version number 999999-SNAPSHOT

Yes, or 2.999999-SNAPSHOT etc. But does it matter practically, if you are just running tests from it?

it contains the commit and not the git tag

Yes, as designed. PCT will check out the commit, which corresponds exactly to the release.

olamy · 2022-11-01T00:09:24Z

the tagged source code (e.g the pom) could have a version number 999999-SNAPSHOT

Yes, or 2.999999-SNAPSHOT etc. But does it matter practically, if you are just running tests from it?

Oh yes good point!! my bad 🤦 we don't care in this case.

it contains the commit and not the git tag

Yes, as designed. PCT will check out the commit, which corresponds exactly to the release.

olamy · 2022-11-09T08:54:09Z

@imonteroperez thanks for approval. but still draft and need a bit more work ;)

olamy · 2022-11-09T09:09:35Z

FYI I need to push a fix I have locally to get this work with the cd releases.
And simplify with using github api or git cli directly

…o not need history here Signed-off-by: Olivier Lamy <olamy@apache.org>

…k/AbstractMultiParentHook.java Co-authored-by: Jesse Glick <jglick@cloudbees.com>

Signed-off-by: Olivier Lamy <olamy@apache.org>

… pct was failing... Signed-off-by: Olivier Lamy <olamy@apache.org>

Signed-off-by: Olivier Lamy <olamy@apache.org>

… not a tag Signed-off-by: Olivier Lamy <olamy@apache.org>

…t history for unit tests Signed-off-by: Olivier Lamy <olamy@apache.org>

Signed-off-by: Olivier Lamy <olamy@apache.org>

olamy · 2022-11-30T21:11:43Z

Blocking pending a draft PR in https://github.com/jenkinsci/bom/pulls (use an incremental version of this PR, and mark https://github.com/jenkinsci/bom/labels/full-test).

done jenkinsci/bom#1613 but I don't have karma to add label

addressed (downstream jenkinsci/bom passes)

Signed-off-by: Olivier Lamy <olamy@apache.org>

jglick · 2022-12-12T14:38:06Z

https://github.com/jenkinsci/bom/pull/1613/checks?check_run_id=10035728045 currently failing?

plugins-compat-tester/src/main/java/org/jenkins/tools/test/PluginCompatTester.java

jglick · 2022-12-12T14:40:14Z

plugins-compat-tester/src/main/java/org/jenkins/tools/test/PluginCompatTester.java

+    }
+
+    public void cloneFromSCM(PomData pomData, String name, String version, File checkoutDirectory, String tag, Map<String, Object> beforeCheckout) throws IOException {
+	    String scmTag = !("".equals(tag)) ? tag : getScmTag(pomData, name, version);


Suggested change

String scmTag = !("".equals(tag)) ? tag : getScmTag(pomData, name, version);

String scmTag = !tag.isEmpty() ? tag : getScmTag(pomData, name, version);

why not if we don't care about being NPE free.

Is tag supposed to be @CheckForNull? For that matter, is "" actually a legitimate argument?

I don't know all the history/reasons of this code so I have just avoid changing it. (except avoid a possible NPE).

I guess StringUtils.isNotEmpty(tag)) will do the job

plugins-compat-tester/src/main/java/org/jenkins/tools/test/hook/AbstractMultiParentHook.java

plugins-compat-tester/src/main/java/org/jenkins/tools/test/PluginCompatTester.java

plugins-compat-tester/src/main/java/org/jenkins/tools/test/hook/AbstractMultiParentHook.java

Signed-off-by: Olivier Lamy <olamy@apache.org>

olamy · 2022-12-13T08:55:13Z

https://github.com/jenkinsci/bom/pull/1613/checks?check_run_id=10035728045 currently failing?

https://ci.jenkins.io/blue/organizations/jenkins/Tools%2Fbom/detail/PR-1613/5/tests/

not sure maybe git and git-client plugins integrating the changes to run in shallow clone are not up2date in this weekly bom

jglick · 2022-12-13T16:52:58Z

maybe git and git-client plugins integrating the changes to run in shallow clone

Is there some PR to fix git-client? @MarkEWaite linked to one for git but said git-client would be more work.

MarkEWaite · 2022-12-13T17:27:24Z

Git client plugin 3.13.1 released 9 days ago with support for testing in shallow cloned repositories. It is included in the bom release 1742.vb_70478c1b_25f. Git client plugin 3.13.1 requires Jenkins 2.346.3 or newer. Either an exclusion will need to be made for 2.332.x or I'll need to backport that test change to a patch version that can support 2.332.x or this will need to wait until the BOM drops support for 2.332.x

jglick · 2022-12-13T17:43:51Z

Ah, jenkinsci/git-client-plugin#944 was not previously linked from here that I noticed.

jglick · 2022-12-13T17:44:28Z

backport that test change

Would be nice. Looks pretty simple right?

MarkEWaite · 2022-12-13T18:18:31Z

I will see what it takes to backport

https://issues.jenkins.io/browse/JENKINS-70174 notes that the git client plugin automated tests failed when using a shallow clone (depth == 1) of a single branch. The issue was fixed for Jenkins 2.346.3 and newer with the release of git client plugin 3.13.1. This release is a backport of that fix. No plugin functionality was changed in this release. It changes the automated tests when run from a shallow clone (depth == 1) of a single branch. jenkinsci#1613 is the key consumer in the plugin bill of materials. jenkinsci/plugin-compat-tester#382 is the key consumer in the plugin compatibility tester.

https://issues.jenkins.io/browse/JENKINS-70174 notes that the git client plugin automated tests failed when using a shallow clone (depth == 1) of a single branch. The issue was fixed for Jenkins 2.346.3 and newer with the release of git client plugin 3.13.1. This release is a backport of that fix. No plugin functionality was changed in this release. It changes the automated tests when run from a shallow clone (depth == 1) of a single branch. #1613 is the key consumer in the plugin bill of materials. jenkinsci/plugin-compat-tester#382 is the key consumer in the plugin compatibility tester.

Signed-off-by: Olivier Lamy <olamy@apache.org>

olamy · 2023-01-18T22:25:03Z

closing this as I have lost motivation to trying to get this merged.....

basil · 2023-01-18T23:00:05Z

Would you regain the motivation if, after merging in most of these changes in #404, I resolved conflicts (making this PR only a few lines of added code) and committed to timely responses?

olamy · 2023-01-19T00:08:18Z

@basil feel free to do whatever you want the branch is still here in a fork https://github.com/olamy/plugin-compat-tester/tree/checkout-shallow as you copy over some changes in another PR I guess re activated shallow clone will be easy.
I'm just demotivated by some nit comments....

basil · 2023-01-19T00:22:51Z

I understand. I can assure you that any feedback left by me would be actionable. I do not intend to pick up the change myself, but the door is always open if you want to pick up where you left off with me as the reviewer.

olamy requested a review from a team as a code owner October 28, 2022 04:23

olamy added the enhancement label Oct 28, 2022

olamy force-pushed the checkout-shallow branch from 53e10fe to 30010cb Compare October 28, 2022 04:25

olamy commented Oct 28, 2022

View reviewed changes

pom.xml Outdated Show resolved Hide resolved

olamy commented Oct 28, 2022

View reviewed changes

plugins-compat-tester/pom.xml Outdated Show resolved Hide resolved

jglick reviewed Oct 28, 2022

View reviewed changes

plugins-compat-tester/pom.xml Outdated Show resolved Hide resolved

plugins-compat-tester/src/main/java/org/jenkins/tools/test/hook/AbstractMultiParentHook.java Outdated Show resolved Hide resolved

olamy requested a review from jglick October 29, 2022 03:13

jglick requested review from imonteroperez and removed request for jglick October 29, 2022 19:50

olamy marked this pull request as draft October 29, 2022 23:40

imonteroperez approved these changes Nov 9, 2022

View reviewed changes

olamy and others added 10 commits November 18, 2022 07:51

upgrade maven scm version to last 1.13.0, use a shallow clone as we d…

217eb6e

…o not need history here Signed-off-by: Olivier Lamy <olamy@apache.org>

Update plugins-compat-tester/src/main/java/org/jenkins/tools/test/hoo…

96a2d01

…k/AbstractMultiParentHook.java Co-authored-by: Jesse Glick <jglick@cloudbees.com>

well checkout of mono module was somewhere else..

29ed279

Signed-off-by: Olivier Lamy <olamy@apache.org>

need something different with cd releases

9877bed

Signed-off-by: Olivier Lamy <olamy@apache.org>

fix upperbound

527adf5

Signed-off-by: Olivier Lamy <olamy@apache.org>

test this as well, but anyway those junit test did not even fail when…

9234241

… pct was failing... Signed-off-by: Olivier Lamy <olamy@apache.org>

using github-api will not help, jgit via maven scm neither

c459bc1

Signed-off-by: Olivier Lamy <olamy@apache.org>

fix clone of jep 229, pom file contains a scm/tag element but this is…

bbbd86d

… not a tag Signed-off-by: Olivier Lamy <olamy@apache.org>

use a hook to avoid shallow clone for plugins such git* which need gi…

492093a

…t history for unit tests Signed-off-by: Olivier Lamy <olamy@apache.org>

add no shallow clone hook

305d9c5

Signed-off-by: Olivier Lamy <olamy@apache.org>

olamy force-pushed the checkout-shallow branch from 8f64dde to 305d9c5 Compare November 17, 2022 22:25

olamy changed the title ~~upgrade maven scm version to last 1.13.0, use a shallow clone as we do not need history here~~ use a shallow clone as we do not need history here (upgrade maven scm version to last 1.13.0) Nov 18, 2022

olamy added a commit to olamy/bom that referenced this pull request Nov 30, 2022

Use incremental build from jenkinsci/plugin-compat-tester#382

c3841f7

jglick mentioned this pull request Dec 1, 2022

Test pct using shallow clone jenkinsci/bom#1613

Closed

no more need of no shallow clone option

5873527

Signed-off-by: Olivier Lamy <olamy@apache.org>

jglick reviewed Dec 12, 2022

View reviewed changes

olamy added 2 commits December 13, 2022 18:05

simplify

f4aeb8b

Signed-off-by: Olivier Lamy <olamy@apache.org>

fix javadoc...

0c12879

Signed-off-by: Olivier Lamy <olamy@apache.org>

MarkEWaite mentioned this pull request Dec 13, 2022

Pin git client plugin 3.12.4 for Jenkins 2.332.x tests jenkinsci/bom#1643

Merged

6 tasks

olamy added 2 commits December 20, 2022 17:05

using StringUtils.isNotEmpty will make the job here

13931ae

Signed-off-by: Olivier Lamy <olamy@apache.org>

oops

6ebf42b

Signed-off-by: Olivier Lamy <olamy@apache.org>

This was referenced Jan 10, 2023

Flatten Maven project #386

Merged

Bump maven.scm.providers.version from 1.5 to 1.13.0 #393

Closed

This was referenced Jan 12, 2023

Remove usage of Maven SCM API #398

Closed

Remove usage of Maven SCM API #404

Merged

olamy closed this Jan 18, 2023

jglick deleted the checkout-shallow branch January 18, 2023 22:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use a shallow clone as we do not need history here (do not use anymore maven-scm but use more simple and faster direct git cli) #382

use a shallow clone as we do not need history here (do not use anymore maven-scm but use more simple and faster direct git cli) #382

olamy commented Oct 28, 2022

jglick left a comment

olamy commented Oct 29, 2022

olamy commented Oct 29, 2022

olamy commented Oct 29, 2022 •

edited

Loading

jglick commented Oct 31, 2022

olamy commented Nov 1, 2022

olamy commented Nov 9, 2022

olamy commented Nov 9, 2022

olamy commented Nov 30, 2022

jglick commented Dec 12, 2022

jglick Dec 12, 2022

olamy Dec 13, 2022

jglick Dec 13, 2022

olamy Dec 16, 2022

olamy Dec 20, 2022

olamy commented Dec 13, 2022

jglick commented Dec 13, 2022

MarkEWaite commented Dec 13, 2022 •

edited

Loading

jglick commented Dec 13, 2022

jglick commented Dec 13, 2022

MarkEWaite commented Dec 13, 2022

olamy commented Jan 18, 2023

basil commented Jan 18, 2023

olamy commented Jan 19, 2023

basil commented Jan 19, 2023

	String scmTag = !("".equals(tag)) ? tag : getScmTag(pomData, name, version);
	String scmTag = !tag.isEmpty() ? tag : getScmTag(pomData, name, version);

use a shallow clone as we do not need history here (do not use anymore maven-scm but use more simple and faster direct git cli) #382

use a shallow clone as we do not need history here (do not use anymore maven-scm but use more simple and faster direct git cli) #382

Conversation

olamy commented Oct 28, 2022

jglick left a comment

Choose a reason for hiding this comment

olamy commented Oct 29, 2022

olamy commented Oct 29, 2022

olamy commented Oct 29, 2022 • edited Loading

jglick commented Oct 31, 2022

olamy commented Nov 1, 2022

olamy commented Nov 9, 2022

olamy commented Nov 9, 2022

olamy commented Nov 30, 2022

jglick commented Dec 12, 2022

jglick Dec 12, 2022

Choose a reason for hiding this comment

olamy Dec 13, 2022

Choose a reason for hiding this comment

jglick Dec 13, 2022

Choose a reason for hiding this comment

olamy Dec 16, 2022

Choose a reason for hiding this comment

olamy Dec 20, 2022

Choose a reason for hiding this comment

olamy commented Dec 13, 2022

jglick commented Dec 13, 2022

MarkEWaite commented Dec 13, 2022 • edited Loading

jglick commented Dec 13, 2022

jglick commented Dec 13, 2022

MarkEWaite commented Dec 13, 2022

olamy commented Jan 18, 2023

basil commented Jan 18, 2023

olamy commented Jan 19, 2023

basil commented Jan 19, 2023

olamy commented Oct 29, 2022 •

edited

Loading

MarkEWaite commented Dec 13, 2022 •

edited

Loading