ubuntu-16.04: unattended updates locking dpkg

[tpetchel]

Test Kitchen fails to converge bento/ubuntu-16.04 version 2.2.7 when multiple CPUs are specified in .kitchen.yml.

The error message is:

dpkg: error: dpkg status database is locked by another process

Repro steps

step 1. Create a cookbook. For example:

$ chef generate cookbook ubuntu-test

step 2. Modify .kitchen.yml to specify 2 CPUs.

---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: ubuntu-16.04
    driver:
      customize:
        cpus: 2

suites:
  - name: default
    run_list:
      - recipe[ubuntu-test::default]
    attributes:

step 3. Run kitchen converge

Result

https://gist.github.com/tpetchel/0b38bb72f296e330c4d90d5c002a63b7

More info

This was reported on Learn Chef: https://learn.chef.io/manage-a-web-app/ubuntu/run-your-web-application-on-a-node/#comment-2717043449.

After seeing the user's .kitchen.yml, I was able to repo on an empty cookbook. The issue does not repro when you do not specify the customize section. That's not to say that it's strictly caused by running more than 1 CPU, but perhaps the timing with multi CPUs just happens to make it repro more consistently.

[mjuarez]

I'm experiencing the same issue however I've manually set the platform's driver to use 1 cpu.

Relevant output:
https://gist.github.com/mjuarez/ac6bfff8b140ad82399ec93549b5faf1

[cheeseplus]

Thanks for the report @mjuarez - when we this was initially reported I expected the CPU count was probably unrelated and more an issue of timing which this seems to support.

[cblecker]

+1 to @mjuarez. Am seeing the same thing with 16.04, using 2.2.7 of the bento box with 1 CPU (no customize section).

[jrb]

This appears to be caused by a combination of an unattended-upgraded package policy set by the Debian/Ubuntu preseed, and a change in how unattended upgrades are handled.

In past versions of Ubuntu, unattended package upgrades were handled by creating a cron job in /etc/cron.daily responsible for running apt. With 16.04 and the shift to systemd, unattended upgrades are handled by a systemd unit. The new systemd unit has OnCalendar set to 6am and 6pm, along with Persistent set to true. The result is that a full apt-get update && apt-get ugrade will run the first time the box boots, in order to "catch-up" with missed runs.

The idea that packages will be upgraded automatically twice a day in my development VMs is a little scary, so I propose disabling the automatic apt upgrades entirely. #612

[hrak]

Since #612 is merged now, can the affected boxes please be updated so they include this fix?

Btw: i'm not sure that #612 fixes the entire issue. There is also a service and a timer in systemd on Ubuntu 16.04 for updating at boot time that may need to be disabled:

sudo systemctl disable apt-daily.service # disable run when system boot
sudo systemctl disable apt-daily.timer   # disable timer run

Kudo's to this page for that

[rmoriz]

@hrak can you check which package provides the unit files?

UPDATE:

apt-daily.service calls /usr/lib/apt/apt.systemd.daily from the apt package and has the following "kill switch":

# check if the user really wants to do something
AutoAptEnable=1  # default is yes
eval $(apt-config shell AutoAptEnable APT::Periodic::Enable)

if [ $AutoAptEnable -eq 0 ]; then
    exit 0
fi

The variable is not set (=default, enable):

# This file understands the following apt configuration variables:
# Values here are the default.
# Create /etc/apt/apt.conf.d/10periodic file to set your preference.
#
#…
#
#  APT::Periodic::Enable "1";
#  - Enable the update/upgrade script (0=disable)

So I guess we'll need /etc/apt/apt.conf.d/10periodic with APT::Periodic::Enable "0"; in it?

[jrb]

Whoops, I guess #612 only disabled the upgrade portion, not the accompanying update, speeding it up just enough to pass in my tests.

It looks like @rmoriz has the right fix -- seems to do the job locally.

[geerlingguy]

I'm testing this for my `geerlingguy.ubuntu1604 box (see: geerlingguy/packer-boxes#7). I'll see how it goes.

[guusdk]

Pending a fix, is there a workaround for this problem? I've tried disabling the unattended upgrade through provisioning, but that fails as the upgrade is already running by the time provisioning kicks in.

[geerlingguy]

@guusdk - the fix seems to be working for my downstream box; it just needs to be added to bento: geerlingguy/packer-boxes@0bcad82

[guusdk]

@geerlingguy That sounds great, but until that happens: workaround? :)

[geerlingguy]

@guusdk - either switch to geerlingguy/ubuntu1604 for now, or you'll need to build your own box with Packer—I couldn't find any way to skip the automatic updates early enough. Although you could probably add a shell provisioner that just waits for a minute or two before your normal provisioner runs.

[loren]

@guusdk You can use this as your chef_omnibus_url in .kitchen.yml: https://gist.github.com/loren/80d42f115fedbdd7635489ecbcfeb121

Context: https://learn.chef.io/manage-a-web-app/ubuntu/run-your-web-application-on-a-node/#comment-2717043449

[cheeseplus]

Testing out the fix now but we seem to still have issues with virtualbox latest version :/

[kaihendry]

Ran into this issue and the chef_omnibus_url from @loren helped me work around it.

Now I'm hitting dpkg lock errors when installing packages.

Thought chef was smarter than this... 😭

[artem-sidorenko]

Following patch did it for me for ubuntu-16.04:

diff --git a/scripts/ubuntu/disable_unattended_updates.sh b/scripts/ubuntu/disable_unattended_updates.sh
new file mode 100644
index 0000000..a2166f3
--- /dev/null
+++ b/scripts/ubuntu/disable_unattended_updates.sh
@@ -0,0 +1,7 @@
+#!/bin/sh -eux
+
+# https://github.com/chef/bento/issues/609#issuecomment-227202607
+cat <<EOF >/etc/apt/apt.conf.d/10disable-auto-apt
+APT::Periodic::Enable "0";
+EOF
+
diff --git a/ubuntu-16.04-amd64.json b/ubuntu-16.04-amd64.json
index 9c0dee8..ce38b67 100644
--- a/ubuntu-16.04-amd64.json
+++ b/ubuntu-16.04-amd64.json
@@ -238,6 +238,7 @@
       "scripts": [
         "scripts/common/metadata.sh",
         "scripts/ubuntu/update.sh",
+        "scripts/ubuntu/disable_unattended_updates.sh",
         "scripts/common/sshd.sh",
         "scripts/ubuntu/networking.sh",
         "scripts/ubuntu/sudoers.sh",

[cheeseplus]

Fixed by #619

[nuest]

I still have this issue - is the fix not yet released?

[Ingramz]

@nuest nope. https://atlas.hashicorp.com/bento

[szeestraten]

Is there a timeline for the next release by any chance?

[cheeseplus]

Bits are being uploaded as I write this - the timeline is rather unfortunately when I have free time because the build pipelines lives at my home.

[geerlingguy]

@cheeseplus - Someday all of us box maintainers should get together and buy a colocated Mac mini or something, and automate the process better :P

That and bandwidth—I know I'm bumping up against 6 TB/month now, and every month or two I have to add on bandwidth.

[cheeseplus]

Up until recently the bandwidth was not a problem as I had symmetric gigabit in Seattle and even a few extra macbook pros. I just moved across the country and returned the extra laptops to my employer so I've definitely been looking at something like Mac Mini Colo.

We do have some automation via buildkite.com where I can batch out the builds across many builders, including linux boxes for vmware and virtualbox (maybe qemu soon), but those boxes are just now being delivered.

[Ingramz]

Is there any reason why this solution can't be used for building the boxes? From the surface it looks exactly what you are trying to do.

[geerlingguy]

@Ingramz - IIRC, that's a for-pay product (bandwidth isn't free)... don't know about whether they offer free bandwidth for OSS boxes though.

[cheeseplus]

@Ingramz several reasons

• Atlas is a pay product.
  • We already get free hosting out of their generosity and we <3<3<3 them for that.
• Atlas doesn't build Parallels so we have to maintain at least some of it ourselves
• We'd already built a lot of this before Atlas was in the current state it is now

There are some other reasons we've had in the past but it's something we've kept in our back pocket for a while waiting to try out again if the cost wasn't an issue.

[Ingramz]

Understandable. The way I see it is that you are going to pay for it one way or another (be it hardware, services or time), but considering they are not displaying pricing information publicly, it might be that you are better off doing this yourself.

[szeestraten]

I see. I thought you guys were already using Atlas.

Perhaps they'd be willing to sponsor the usage as they officially recommend your boxes :)

[rmoriz]

If you just need fast build boxes to automate a workflow, consider https://www.hetzner.de/us/hosting/produkte_rootserver/px121ssd (yes, they are located in Germany, but they have a very good connection and look at the price). They have an API to setup the boxes (https://github.com/rmoriz/hetzner-api https://github.com/rmoriz/hetzner-bootstrap)

[cheeseplus]

We have looked at it and it's one of those "it depends" areas:

• renting a mini isn't bad but the bandwidth is killer even with everything proxied to hell and back
• @rmoriz catch there is we NEED OSX for Parallels :/

[rmoriz]

@cheeseplus OSX: https://macminicolo.net/pricing.html

[cheeseplus]

Yep, I've already talked to the owner personally ;) just a bit more capital outlay than I am capable of authorizing on behalf of my company.

[bflad]

We should probably move the hosting/publishing to its own issue and/or collaboratively work on a document outlining the existing process and potential features/goals moving forward.

In the meantime, how much are we talking? Should we form a non-profit/LLC for this that in turn companies/people can contribute towards so no one has to foot the whole bill? I don't mind using my own LLC for billing purposes if necessary assuming I can re-coup at least the majority of costs.

[MasterJames]

This is happening in 19.04 and is annoying. it also got stuck on a hash padding in an upgrade 0001 file?!

[cheeseplus]

Commenting on a closed issue about a platform that hasn't been published yet with no real detail other than a complaint is not productive or helpful.