Jenkins log and artifact cleanup schedule #1017
Comments
|
I'm ok with this, or with smaller numbers even. Do you know about how many builds consumed the 264G? Perhaps others will have comments. |
|
I checked a full Java 9 build and it was 4G uncompressed. This includes the git repos, all the source, build artifacts, and images. I assume the SDK kept around must be smaller than this. The jdk image itself is 323M and zipped its 202M, so around 20G for 100 builds. |
|
We probably want a bigger number for PRs, "Max # of builds to keep with artifacts", say 100 to match the "Max # of builds to keep". |
|
Biggest offenders currently are the PR Sanity builds. (no cleanuo yet) |
|
Based on how much space we've consumed in ~3 months and suggestions in comments, I have updated the numbers in the description. Propose we keep 2 months/100 builds? |
|
@AdamBrousseau Are there any download stats on how often these artifacts have been downloaded? If they aren't being downloaded, then storing them serves no purpose. |
|
I guess a larger number of days to keep builds doesn't much matter if they are limited to 100 builds. What does it mean not to put a limit on the artifacts? Won't the artifacts eventually consume too much space, or do they disappear with the builds? |
|
Artifacts should disappear with the builds (from Jenkins master). Based on typical use cases, do we realistically need to keep the SDK for 100 builds for PR builds? I had thought the use case was, "as a developer I want to download the SDK from my PR build, immediately after I build it, so I can do some additional local testing. After a few days (x number of builds later), I want to rebuild, as I want to keep current with all of the other changes coming into the repo)." As a developer, do I ever want to use an SDK that is 100 builds old? Remind me, is the Jenkins logic to compare "days to keep" vs "# of builds", and uses the smallest? So pshiptons question above would matter more for slow projects that do not have lots of build activity over the course of y days. |
The artifacts would get cleaned up with the build logs.
I don't see anything built into Jenkins for this. Initial searching only talks about website monitoring services. |
|
Can we close on this issue?
|
|
Are you saying we get 50 builds per job? i.e. 50 for Java 9 pLinux, 50 for Java 9 zLinux, etc. |
|
I'd say make the change, and then we can adjust later as necessary. |
|
Thought about it a bit more. Makes more sense to me to do something like this
Modified all build configs. |
|
Reopening this for discussion as we've got an eclipse bugzilla opened against us for consuming 365G space. They would like us to stay under 100. Some napkin math:
To summarize, Propose we keep the last 10 of every build (PullRequest* or Build*). Disclaimer: This does not account for build logs
I also have a job I can setup to monitor the space we are consuming and list the top 10 "Pig" jobs. Optional slack notifications as well. |
|
Sounds good. If its still too big with the logs we can reduce builds for the PRs. |
|
Agreed. Though I'm fine with decreasing the number of builds kept until there is evidence someone is actually downloading them. |
|
Updated all the build and PR jobs to only keep last 10 artifacts. I will keep an eye on the disk space over the next few days to see if it reduces. |
|
https://bugs.eclipse.org/bugs/show_bug.cgi?id=535057 |
* keep the artifacts on master to a minimum; no archiving on pr build * Issue eclipse-openj9#1017 * [skip ci] Signed-off-by: Joe deKoning <joe_dekoning@ca.ibm.com>
|
Joe delivered a change to stop archiving SDKs in PR builds (#1988) |
[skip ci] Issue eclipse-openj9#1017 Signed-off-by: Adam Brousseau <adam.brousseau88@gmail.com>
|
Only storing 5 SDKs per build now. Usage went from 180G to 173G. |
- Add Build Discarder
- Retrieve values from Variable file
- Add job Parameters
- Default values for Vendor Params must be set globally
even if they are blank
- No Trigger for Build and Pipeline jobs
- TODO add trigger for PR builds
Issue eclipse-openj9#1017
[skip ci]
Signed-off-by: Adam Brousseau <adam.brousseau88@gmail.com>
- Add Build Discarder
- Retrieve values from Variable file
- Add job Parameters
- Default values for Vendor Params must be set globally
even if they are blank
- No Trigger for Build and Pipeline jobs
- TODO add trigger for PR builds
Issue eclipse-openj9#1017
[skip ci]
Signed-off-by: Adam Brousseau <adam.brousseau88@gmail.com>
- Minimally we need to pass ARTIFACTORY_SERVER to the test jobs in order for them to upload test artifacts to Artifactory. - Write the rest of the Artifactory variables to the build env for future use. - We have the option to pass more of these variables in the future if we wish. See - adoptium/aqa-tests#591 - adoptium/aqa-tests#561 - eclipse-openj9#1017 [skip ci] Signed-off-by: Adam Brousseau <adam.brousseau88@gmail.com>
- Delete all Build artifacts after the overall pipeline has completed - Always remove artifacts even if tests fail Related eclipse-openj9#1017 [skip ci] Signed-off-by: Adam Brousseau <adam.brousseau88@gmail.com>
Today we took down Eclipse's hipp5 which is where our OpenJ9 Jenkins master lives. We were consuming 264G space according to the Bugzilla and have been asked to reduce.
What we store:
Jenkins has the following options, which are configured on a per job basis
Currently, our nightly (and OMR acceptance) builds are setup with the following:
Our PR builds are not configured to discard (keep everything).
Propose:
Nightlies and OMR Acceptance:
710PRs:
1425Please review @pshipton
Also, I have a script to check disk space I could add a job for and send a Slack notification if we're over X. Could open as a separate issue if we're interested.
The text was updated successfully, but these errors were encountered: