/etc/hosts gets clobbered

[gavanderhoorn]

This might be "too localised", but I just noticed that /etc/hosts gets clobbered by puppet on all hosts during deployment (and the file gets the standard "don't modify this file any more" header).

According to PUP-2289 and PUP-4383 this should not happen (after puppet was updated to not do that), but afaik, all my VMs had an Ubuntu-standard hosts file, and I've also not seen any lines in puppet.log about this.

[nuclearsandwich]

Following the bugs that you linked it appears that this issue is only fixed in Puppet 4.0 (puppetlabs/puppet#3005 (comment)). We're currently using puppet 3.8.0 from the upstream ubuntu repositories. #146 did switch us to the puppet "future" parser used in puppet 4.x as a step toward that transition.

[gavanderhoorn]

We're currently using puppet 3.8.0

right. I'd missed that.

[nuclearsandwich]

@gavanderhoorn I've ticketed what it would take to move to puppet 4 and fix the bug. It's not something that I think I'll be able to work on for the foreseeable future.

Workaround options

A workaround would be adding other hosts via puppet resources as well. One of the things that I haven't looked into in detail is interop with other puppet modules since anything that OSRF needs gets committed to this repo.

One medium-effort way to do it would be to refactor the hosts code in buildfarm_deployment and buildfarm_deployment_config to pull from a list structure rather than looking in particular for master::ip and repo::ip.

The minimal-effort (for maintainers) is for teams that need to add additional puppet code to add it to the reconfigure script in their fork of buildfarm_deployment_config.

[gavanderhoorn]

Thanks for the info @nuclearsandwich.

Perhaps we can add some note that deploying to hosts that have 'special' DNS setups is currently complicated by this Puppet bug.

At least my hosts started to act up and I couldn't figure out what was going on, until I checked /etc/hosts.

[nuclearsandwich]

Perhaps we can add some note that deploying to hosts that have 'special' DNS setups is currently complicated by this Puppet bug.

That sounds fine to me. Are you imagining a note during the puppet run or something in the documentation for buildfarm_deployment? I would think that most folks don't read the puppet logs unless something goes wrong so documentation would be better.

At least my hosts started to act up and I couldn't figure out what was going on, until I checked /etc/hosts.

According to the referenced puppet issues, the existing hosts file should only be clobbered if puppet is unable to parse it. Is there some particular aspect of your hosts file that puppet is failing to parse which you expect to be common? All of the hosts files for build.ros.org machines appear to have their handspun changes intact.

I think it's perfectly valid to put a warning / known issues list up in the documentation. I'm just curious how wide the impact of this particular issue is.

[gavanderhoorn]

Are you imagining a note during the puppet run or something in the documentation for buildfarm_deployment?

I would say readme. A 'known issues' as you mention would probably be a good place.

re: no one checks the log: I agree. You typically only check it when something went wrong. But even in that case it might not help, as I did not see any mention of a problem with the hosts file.

Is there some particular aspect of your hosts file that puppet is failing to parse which you expect to be common?

I don't know why puppet ate my hosts file. As far as I can tell it's the standard Ubuntu /etc/hosts file, just with the hostnames of my vms in there.

I can see whether I can do a test deployment on a fresh VM to see what triggers it.