Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix qemu container integration #249

Merged
merged 1 commit into from
Dec 12, 2022
Merged

Fix qemu container integration #249

merged 1 commit into from
Dec 12, 2022

Conversation

dbast
Copy link
Collaborator

@dbast dbast commented Dec 11, 2022

packer-builder-arm is using qemu in order to emulate the arm target platform when provisioning arm machine images.

The Docker Container therefor contains embedded qemu binaries and some setup code to register those when using the container. This integration didn't work and that was not caught by the Github workflows due the docker/setup-qemu-action always being running as part of the docker workflow. The usage of docker/setup-qemu-action enables two things: 1. building multi-arch containers 2. running binaries of other platforms inside a container. The later should be covered by the qemu binaries+setup we integrated into the container. By setting up docker/setup-qemu-action only when we build+push multi-arch containers on the master-brach/release, we get the qemu integration tested during PR test builds. So we get that tested, but how to resolve the issue of the not working qemu integration?

This are afaik the most used qemu binaries:

  1. Delivered by the container from https://github.com/tonistiigi/binfmt with huge amount of downloads due to a very wide usage by docker/setup-qemu-action. They maintain there own patches and it also works in our case by using the action or running the container before the packer run. But copying the binaries into our container did not work so far (why?).
  2. Also widely used is https://github.com/multiarch/qemu-user-static, but only provides binaries for x86_64 hosts, which is not sufficient for creating linux-aarch64 containers useable on M1 MACs.
  3. The qemu-user-static packages as part of various distributions. Those work, but the Debian/Ubuntu tool update-binfmts to register the binaries does not register qemu-arm-static in case of a linux-aarch64 host, which leads to packer again not working.

What this PR now does is combining the binaries from 3. with registering also the qemu-arm-static binary on linux-aarch64 like done by 1.

The resulting container should now work on the following platforms

  • on x86_64 Linux (using the integrated qemu, without any previous docker qemu enablement required)
  • on x86_64 macOS (using the integrated qemu, without any previous docker qemu enablement required)
  • on arm64 macOS (Apple M1/M2) (using the integrated qemu, without any previous docker qemu enablement required)
  • on any of the above by disabling the integrated qemu:
    • by enabling docker qemu integration via docker/setup-qemu-action (= locally docker run --privileged --rm tonistiigi/binfmt --install all)
    • running our container via: docker run -e SETUP_QEMU=false --rm --privileged -v /dev:/dev -v ${PWD}:/build --entrypoint /bin/bash -it mkaczanowski/packer-builder-arm

Found by the experiments in #249

cc: @benalexau @openoms who reported this.

fixes: #244
fixes: #245

@dbast dbast marked this pull request as ready for review December 12, 2022 09:46
@dbast dbast merged commit a9af016 into mkaczanowski:master Dec 12, 2022
@dbast dbast deleted the fix_qemu_integration branch December 12, 2022 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant