Openstack cloud provider causing apiserver container to fail

[bputt]

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version): 1.8.4

Environment:

• Kubernetes version (use kubectl version): 1.8.4
• Cloud provider or hardware configuration: openstack icehouse
• OS (e.g. from /etc/os-release): CentOS 7.4
• Kernel (e.g. uname -a): Linux 3.10
• Others: Docker 17.09.0-ce

What happened?

When adding --cloud-provider=openstack and --cloud-config=/etc/kubernetes/cloud.conf to /etc/kubernetes/manifests/kube-apiserver.yaml I get the following error on startup of the container:

[mount_linux.go:142] Mount failed: exit status 1
Mounting arguments: -t vfat -o ro /tmp/configdrive071298910
Output: mount: can't read '/etc/fstab': No such file or directory
Error mounting configdrive: mount failed

What you expected to happen?

For the container to find the config-2 drive and stay alive

How to reproduce it (as minimally and precisely as possible)?

/etc/kubernetes/manifests/kube-apiserver.yaml and /etc/kubernetes/manifests/kube-controller-manager.yaml share the following:

spec:
  containers:
  - command:
    - --cloud-provider=openstack
    - --cloud-config=/etc/kubernetes/cloud.conf
    ...

volumeMounts:
- mountPath: /etc/kubernetes/cloud.conf
  name: cloud-config
  readOnly: true
...

volumes:
- hostPath:
    path: /etc/kubernetes/cloud.conf
    type: File
  name: cloud-config
...

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --cloud-provider=openstack --cloud-config=/etc/kubernetes/cloud.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

/etc/kubernetes/cloud.conf

[Global]
auth-url=...
tenant-id=...
username=...
password=...
region=...
ca-file=...

Anything else we need to know?

The openstack metadata url is not available as HTTP, so unless HTTPS support is added, I need the config drive to work as expected.

Referred here from: kubernetes/kubernetes#47392

[xiaosuiba]

After digging a little into the source code , I found a work around. You just have to map /dev/sr0 device into your container and set the container as privileged.
Here is a working example:

...
volumeMounts:
- mountPath: /dev/disk/by-label/config-2
  name: sr0
...
volumes:
- hostPath:
    path: /dev/sr0
    type: BlockDevice
  name: sr0
securityContext:
 privileged: true

[luxas]

Can you check whether this is fixed in v1.9.0
cc @dims

[dims]

/assign @dims

[dims]

@xiaosuiba no need to mount /dev/sr0 just adding privileges is enough (at least with 1.9.0+), i have proposed a PR to enabled that for OpenStack based kubeadm (see 57561)

Thanks a lot for all your research, it helped quite a bit in fixing this issue.

-- Dims