Introduce "collection" projects for better usage of hierarchical view #2041 #3258

rkg-mm · 2023-12-02T16:18:16Z

Description

This change introduces logic for "collection projects". Those are basically projects used as parent for other projects that shall not hold any own component or vulnerability data, but instead get calculated from child projects using different configurable aggregation logics.

Corresponding Frontend PR with Screenshots: DependencyTrack/frontend#658

Addressed Issue

resolves #2041
resolves #657
resolves #2410

Additional Details

Checklist

I have read and understand the contributing guidelines
~~This PR fixes a defect, and I have provided tests to verify that the fix is effective~~
This PR implements an enhancement, and I have provided tests to verify that it works as intended
This PR introduces changes to the database model, and I have added corresponding update logic
This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

cheapshot2000 · 2024-03-25T21:52:24Z

Has there been any updates/progress on the feature in completion of the 4.11 release?

TWpgo · 2024-04-22T11:34:35Z

Looking forward for this feature 👍

rkg-mm · 2024-04-23T08:56:17Z

@nscuro I wonder why the unit test still fails. I don't see a change that could influence the parent of a project not being serialized?

nscuro · 2024-04-23T09:10:50Z

@rkg-mm FWIW there can be subtle changes in how serialization libraries (i.e. Jackson) handle certain fields, when those libraries are updated. Or how the persistence framework fetches fields implicitly.

rkg-mm · 2024-04-23T09:33:16Z

Yea so, the PRs backend & frontend seem to be working well. But I have no clue about this unit test and will need some help here. The test explicitly tests the now not working condition, so I guess its important that it works, but I don't see a change that causes the failure.

 Error:  Tests run: 40, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 74.367 s <<< FAILURE! - in org.dependencytrack.resources.v1.ProjectResourceTest
Error:  patchProjectParentTest(org.dependencytrack.resources.v1.ProjectResourceTest)  Time elapsed: 1.483 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: 
JSON documents are different:
Different keys found in node "", missing: "parent", expected: 
<{"active":true,"collectionLogic":"NONE","name":"DEF","parent":{"name":"GHI","uuid":"${json-unit.matches:parentProjectUuid}","version":"3.0"},"properties":[],"tags":[],"uuid":"${json-unit.matches:projectUuid}","version":"2.0"}> 
but was: 
<{"active":true,"collectionLogic":"NONE","name":"DEF","properties":[],"tags":[],"uuid":"ba033fa5-aa20-44ae-85de-6a33e5cf36a3","version":"2.0"}>

	at org.dependencytrack.resources.v1.ProjectResourceTest.patchProjectParentTest(ProjectResourceTest.java:733)

nscuro · 2024-04-28T13:01:51Z

src/main/java/org/dependencytrack/model/ProjectCollectionLogic.java

+ * @author Ralf King
+ * @since 4.11.0
+ */
+public enum ProjectCollectionLogic {


It would be good to add some documentation about this. In #2041 you mention a few use cases that these address, having those be part of the docs will definitely help others.

Docs added. However, since I'm stuck to Windows and there is no build script for the docs in windows, I could not really test if this integrates properly. Can you check this?

src/main/java/org/dependencytrack/persistence/ProjectQueryManager.java

nscuro · 2024-04-28T13:24:06Z

src/main/java/org/dependencytrack/persistence/ProjectQueryManager.java

@@ -538,6 +547,7 @@ public Project updateProject(Project transientProject, boolean commitIndex) {
        project.setDescription(transientProject.getDescription());
        project.setVersion(transientProject.getVersion());
        project.setClassifier(transientProject.getClassifier());
+        project.setCollectionLogic(transientProject.getCollectionLogic());


Changing the collection logic from say AGGREGATE_DIRECT_CHILDREN to HIGHEST_SEMVER_CHILD can drastically change metric values on existing projects. There will be no indication as to if and when the logic was changed, which can make it tricky to explain why values changed so much.

That's correct, but also it would be an intended change. Idk what could be done about this except add such change to the logfile, but that would only be helpful for deep analysis of such issue.

Fair.

Naive idea: We could add a Collection logic change flag to ProjectMetrics. We could use that in the metrics graph, in some similar way to what you can do in Grafana with annotations:

I like that idea, but at least on a first look the frontend chart lib doesn't support this. not sure how difficult this would be to add on top of it. Any experience with that? I never used that component.
From data perspective in backend it should not be too complicated.

edit: or would be adding it to the charts tooltip be enough? Also it seems there are options to style datapoints differently. Maybe we could give such points a different styling + add the info in the tooltip?

I don't have too strong opinions on how we highlight it, but I do think that highlighting it one way or the other would be good.

Tooltip on it's own might be too hidden, but styling the datapoints sounds good.

@nscuro would you be fine with creating a new issue to add this visibility later as further enhancement, and for now rely on log file for visibility in case someone wonders? we could also add a hint in the project edit UI when someone changes the logic, that informs him of this for now?

Logging is probably fine for an MVP implementation, yes. But ultimately we will really need some kind of indicator in the UI.

@nscuro I found a way to show it in the metrics chart. Only issue is that the line gets invisible when hovered. No Idea why. But I think this is fine as it is, looks like its intended :D

Ok found the reason for the visibility issue and fixed it. Also extended it to the other charts now. Looks like this:

Hint: The blue line at 0 will no longer be there, I improved the backend to not record the initial entry after an update to the new version anymore as changed collection logic, my DB just has these entries already

src/main/java/org/dependencytrack/tasks/metrics/ProjectMetricsUpdateTask.java

src/main/java/org/dependencytrack/model/ProjectCollectionLogic.java

nscuro · 2024-04-28T14:07:38Z

@rkg-mm But I have no clue about this unit test and will need some help here. The test explicitly tests the now not working condition, so I guess its important that it works, but I don't see a change that causes the failure.

I debugged the area in question multiple times now and still don't quite understand how this is happening. The parent is part of the response when I comment out these lines: https://github.com/DependencyTrack/dependency-track/pull/3258/files#diff-75cd29a0a085d84dff3cb4794242e0d1b400385f1acfae172346c3e6780629b0R582-R590

Something about those lines causes parent to be unloaded. But they're literally just calling a setter, so I don't quite get why. The test succeeds when adding project.getParent(); after the lines mentioned above. This kind of "force loading" is used in other areas of the code base. I don't like it, but in this case I think it's OK to do it.

rkg-mm · 2024-05-05T20:46:36Z

oh come on.. Idk what IntelliJ is doing but whenever I rebase this branch on your master I end up having all changes from master in my PR again. What am I doing wrong here? I don't get it...

edit: ok fixed it again with some hacky workaround. still don't know what I do wrong here...

nscuro · 2024-05-05T20:54:08Z

@rkg-mm How do you do the rebase? I usually run git pull --rebase upstream master in the command line and then use IntelliJ to resolve any conflicts in case there are any.

I never had this particular problem happen, but can imagine IntelliJ might do some unexpected things when merging/rebasing via UI.

rkg-mm · 2024-05-05T20:58:58Z

In IntelliJ UI I select the upstream master and select "Rebase (my branch) onto master" then push the changes to my remote. The strange thing happens here already, as suddenly my own remote has changes not recognized by my local branch and needs to either be merged (which I tried before and ended up with double changes on same files) or I select "rebase anyway" which avoids double merging. But in both cases I still end up with all those unwanted changes again. Maybe I should try the console attempt next time...

rkg-mm · 2024-05-06T16:16:16Z

@rkg-mm But I have no clue about this unit test and will need some help here. The test explicitly tests the now not working condition, so I guess its important that it works, but I don't see a change that causes the failure.

I debugged the area in question multiple times now and still don't quite understand how this is happening. The parent is part of the response when I comment out these lines: https://github.com/DependencyTrack/dependency-track/pull/3258/files#diff-75cd29a0a085d84dff3cb4794242e0d1b400385f1acfae172346c3e6780629b0R582-R590

Something about those lines causes parent to be unloaded. But they're literally just calling a setter, so I don't quite get why. The test succeeds when adding project.getParent(); after the lines mentioned above. This kind of "force loading" is used in other areas of the code base. I don't like it, but in this case I think it's OK to do it.

Fixed by your suggestion

codacy-production · 2024-05-06T16:36:32Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.10% (target: -1.00%)	✅ 86.04% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`ff22d4f`)	22069	16756	75.93%
Head commit (`0cfdd8c`)	22259 (+190)	16923 (+167)	76.03% (+0.10%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3258)	222	191	86.04%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy will stop sending the deprecated coverage status from June 5th, 2024. Learn more}

nscuro · 2024-05-07T14:12:39Z

@rkg-mm We have to schedule this for 4.12 to allow us for sufficient time to review and test.

rkg-mm · 2024-05-07T16:04:35Z

@rkg-mm We have to schedule this for 4.12 to allow us for sufficient time to review and test.

no problem, I understand. Lets just hope the release cycle is a bit shorter this time :D

cheapshot2000 · 2024-05-07T16:06:22Z

@rkg-mm We have to schedule this for 4.12 to allow us for sufficient time to review and test.

no problem, I understand. Lets just hope the release cycle is a bit shorter this time :D

Is there a set release cadence? or other logic for determining when releases are completed?

nscuro · 2024-05-07T16:18:09Z

Is there a set release cadence? or other logic for determining when releases are completed?

There is not at the moment. We originally planned to do monthly releases, but ended up stretching that (by far...) yet again. A problem is that the core team has certain items they want (and have to work on) in a given release, while said core team also needs to review and test community contributions. It's a luxury problem, but it means that more time is needed to meet both our own, and the community's expectations.

This particular PR was planned for inclusion in 4.11 for a long time, but it also touches on the very core of how DT behaves, in turn requiring more review and testing than other, smaller PRs. And of course contributors also don't always have time to answer our annoying requests. :)

We're still in the process of figuring out how to best plan and size releases. Going forward, we're labelling issues with T-shirt sizes, according to their effort. Aim being to limit the number of high-effort items per release, enabling faster cycles.

msymons · 2024-08-08T08:26:47Z

Re-assigning to 4.13 milestone in order to reduce the pressure on contributors to get this finished whilst allowing v4.12.0 to be released quicker.

codacy-production · 2024-09-14T16:37:29Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.09% (target: -1.00%)	✅ 88.74% (target: 70.00%)

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`6f7c49c`)	22460	17770	79.12%
Head commit (`2bf56f0`)	22649 (+189)	17941 (+171)	79.21% (+0.09%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3258)	222	197	88.74%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

rkg-mm · 2024-09-16T16:31:54Z

@nscuro apart from the one proposal in the UI PR which I added some question for, this should be finished now. Can you review it again please?

netomi · 2024-10-02T18:19:08Z

Nice to see this PR moving forward. We run a previous version of the PR in our experimental setup. Will update with the latest version and provide feedback.

Some minor thing that we noticed and that would be great if that could be changed or supported as well: when modeling your products in a parent / child relationship like that:

You are forced to set a classifier for the parent project, though it is purely a project container and the classifier does not make any sense in this context. If the classifier could be made optional or better reflect the concept of a project container, it would be nice and avoid confusion when looking at the project hierarchy.

rkg-mm · 2024-10-03T09:55:48Z

Nice to see this PR moving forward. We run a previous version of the PR in our experimental setup. Will update with the latest version and provide feedback.

Some minor thing that we noticed and that would be great if that could be changed or supported as well: when modeling your products in a parent / child relationship like that:

You are forced to set a classifier for the parent project, though it is purely a project container and the classifier does not make any sense in this context. If the classifier could be made optional or better reflect the concept of a project container, it would be nice and avoid confusion when looking at the project hierarchy.

So basically you want to have an option which marks the parent as a collection project, hence removes the component uploads etc. but does not calculate any values based on children? what should be displayed inside the parent then? nothing?

…CollectionLogics. Modifies project metrics updates to consider collection project logics. Added several events to trigger project metrics updates to update collection projects if child projects change in a relevant way. Signed-off-by: Ralf King <rkg@mm-software.com>

Add test cases for new metrics update logic for collection projects. Signed-off-by: Ralf King <rkg@mm-software.com>

… all projects Signed-off-by: Ralf King <rkg@mm-software.com>