-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor for deploying a Xenial buildfarm. #146
Refactor for deploying a Xenial buildfarm. #146
Conversation
In an offline discussion earlier this week I decided that testing with and supporting a single node buildfarm was not going to be a goal for this PR. The refactoring I've done so far should make it much more doable but right now I want to focus on getting the official buildfarm tested and then migrated to Xenial. Once that's complete circling back to support a single node configuration can be done with time pressure. It's still something we want to do as it greatly eases running local or alternate buildfarms on smaller setups. |
I also found and fixed some bugs related to the systemd <-> initscript interaction for the agent profile which was blocking further work on the repo profile. |
d5dc35d
to
c48f585
Compare
This PR is completely untested on Trusty and I don't currently see a reason to try and achieve trusty compatibility. Plugin dependency management and versioning is still a royal pain so there will be some post setup manual work to do. As this is my first time migrating a ROS buildfarm I'll be looking for help preparing a test and migration checklist. |
👍 Supporting only Xenial is absolutely fine.
For a first steps that might be fine. But in order to use this for CI testing it needs to be fully automated at some point.
I guess the "best" test is setting up a test farm and let it build e.g. all of Lunar and compare that the jobs which pass on the live farm also pass on the test farm. |
Do you know how this was dealt with previously? The puppet module for Jenkins doesn't resolve plugin dependencies because only the latest (non-LTS even) plugin versions have dependency metadata provided via the Jenkins wiki. There is an effort underway at rtyler/jpm to get a sophisticated package manager for Jenkins plugins off the ground but it's also in very early stages and doesn't yet support LTS. EDIT: I should point out that this step is currently necessary using the master configs on Trusty, not a regression caused by the refactor / updates. It's definitely something I tried to resolve, but there doesn't seem to be a solution beyond pinning every plugin version in the config, then manually adding each plugins recursive dependencies to the puppet configs, which seems like a lot of work to redo whenever we update plugin versions. |
The puppet files explicitly installed the dependencies. Those lines usually had a comment to mention which other plugin they are required for. While that is certainly more effort it avoids any manual work for deploying the machines. Once a better solution is available we should certainly change to that to avoid having to list the dependencies explicitly. |
cc50aca spikes out a script that reads a current Jenkins instance and dumps the currently installed plugins as an include-able puppet module. Each plugin has a require block for it's dependencies rather than a comment with its dependents since that's the format of the data from Jenkins but we could easily invert that if we think it's preferable. |
Thanks for all this good work. I'm looking to set up a small (4-5 machines) build farm, do you suggest trying to use this branch or sticking to trusty for now? |
If you're familiar with the ROS buildfarm's design (I wasn't when I started this project 😁), plan to keep your farm around for a while, and don't mind a few speedbumps early in the setup process, then the Xenial branch is probably suitable. I think the prime advantage of using Xenial is Java 8 for the Jenkins master. Since the packages themselves are built in containers, the host OS has less impact on available dependencies. If you're going to be making modifications to the puppet code, the refactor here is intended to make it much more grok-able and customizable but there's still work to be done documenting recommended ways to hook in and layer this code into your own puppet scripts. I particularly need to document the hiera keys. The trusty code has been configuring agent machines consistently for the current farm though the plugin versions and dependencies are out of sync with what's currently deployed on the farm (which can be resolved by updating plugins in the Jenkins UI). If you want to take advantage of that proofing or your farm needs to match as faithfully as possible to the currently deployed ROS buildfarm then the code in master is the place to start, with the caveat that we are in early testing for a xenial based buildfarm and want to migrate the current farm in the coming months. Be sure to browse some of the recent issues on this repository, particularly #149 as there are some problems with both the current master and the current xenial branch. |
c14135d will need to be incorporated here. |
I didn't see any Java arguments in the patch but I might have just missed them. What |
For now, the Jenkins Java arguments are still part of the hiera config rather than the Puppet. This is how they were maintained previously and I was changing so much else that I wanted to leave anything I didn't have an explicit reason to change the way it was. I need to take time to read through the entire referenced blog post still. It looks like a lot of the flags they're using are tailored toward logging and diagnostics. Is there going to be a disk utilization (storage and throuput) consideration if we just enable all of those? Are there tuning flags we should adopt directly without the logging for now? |
That's why I didn't see them in the diff 😉 Thanks for the pointer.
I wouldn't blindly enable all the arguments mentioned in the blog post. Since we didn't have Java 8 I wasn't able to try those but the config file on the existing Jenkins machine has the options I thought would be most valuable in a commented line:
|
Let's move the JVM tuning discussion to the config repo in order to keep the discussion closer to the code. |
I have changed the target branch of this pull request to point to a new |
Xenial uses systemd so we'll need to change how the explicit upstart code is used.
* Remove plugin declaration (already declared?) * Specify full python version. * Try alternate pip installation allocation. * Explicitly declare curl package.
I put this in just to test that it resolved the issue, now I want to justify its presence.
The 2.0 Docker python library removes untagged ('<none>:<none>') images from the RepoTags image attribute so this should test for an empty list rather then an explict none tag. https://github.com/docker/docker-py/blob/2.4.2/docker/models/images.py#L41-L44 The remove_image function was also moved and should be accessed by the client's `image` attribute.
docker-py appears to be an abandoned copy of the docker pypi package stuck at 1.10.6. An issue with the script on Xenial prompted us to update the dependency.
I think we'll want to start using the Jenkins CLI for managed configuration of plugins. It's the recommended config management solution for multiple plugins and if I can figure out how to use the Jenkins puppet module to run Groovy script files rather than inline scripts we'll be even better off. This change resolves ros-infrastructure#148
There's more documentation QA to be done but I think I'll do that as part of a broader docs sweep after adding a solo-machine buildfarm.
At some point this should be cleaned up, factored out, and released as a separate puppet module. Can't be bothered now though.
07b5598
to
2459d0f
Compare
Refactor for deploying a Xenial buildfarm.
This is a heavy refactor of the buildfarm deployment puppet.
The purpose of the refactor is to
Outstanding work
Allow a single node to have multiple roles make repo, master and slave non colliding #42(deferred)Issues resolved by this PR
Outside the scope of this PR is completing the move to Puppet 4. Some of the modules we're using don't yet explicitly support Puppet 4 and Xenial ships with the recently EOL'd 3.8.5 which is what this targets currently. We are using the "future" parser which is shared by puppet 3 and 4. I've also incorporated a puppet lint config and I think it'd be a nice idea to get puppet-lint and
puppet parser validate
running through travisci or something.This PR is quite large and moves a lot of stuff around. When it comes time to review I'll point out a few highlights and areas I think need the most attention.