DNS timeout in 2.13.0 with flannel/canal on rhel7

[HugoRh]

Environment:

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  Linux 3.10.0-957.27.2.el7.x86_64 x86_64
  NAME="Red Hat Enterprise Linux Server"
  VERSION="7.6 (Maipo)"
  ID="rhel"
  ID_LIKE="fedora"
  VARIANT="Server"
  VARIANT_ID="server"
  VERSION_ID="7.6"
  PRETTY_NAME="Red Hat Enterprise Linux"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
  HOME_URL="https://www.redhat.com/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"

• Version of Ansible (ansible --version):
  ansible 2.9.6
  config file = /data/caas/deploy-k8s/kubespray/ansible.cfg
  configured module search path = [u'/data/caas/deploy-k8s/kubespray/library']
  ansible python module location = /opt/rh/python27/root/usr/lib/python2.7/site-packages/ansible
  executable location = /opt/rh/python27/root/usr/bin/ansible
  python version = 2.7.16 (default, Jun 28 2019, 16:57:28) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
• Version of Python (python --version):
  Python 2.7.16

Kubespray version (commit) (git rev-parse --short HEAD):
01dbc90

Network plugin used:
canal

Anything else do we need to know:
I upgraded my test cluster from 2.12.5 to 2.13.0 without issue.
Yet , soon after my cluster was available again, many pods were in CrashLoopBackOff status.
And I noticed pretty quickly some issue at the DNS level with the nodelocaldns pods:

 on 169.254.25.10
cluster.local.:53 on 169.254.25.10
in-addr.arpa.:53 on 169.254.25.10
ip6.arpa.:53 on 169.254.25.10
[INFO] plugin/reload: Running configuration MD5 = adf97d6b4504ff12113ebb35f0c6413e
CoreDNS-1.6.7
linux/amd64, go1.11.13, 
[ERROR] plugin/errors: 2 1643225653471384235.1396585009018888177.cluster.local. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 2209263729514313689.8562382610265883181.in-addr.arpa. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 3642054395277035928.1497406778482370717.ip6.arpa. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 1643225653471384235.1396585009018888177.cluster.local. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 3642054395277035928.1497406778482370717.ip6.arpa. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 2209263729514313689.8562382610265883181.in-addr.arpa. HINFO: dial tcp 10.233.0.3:53: i/o timeout
[ERROR] plugin/errors: 2 my-jaeger-agent.observability.svc.cluster.local.kube-system.svc.cluster.local. AAAA: dial tcp 10.233.0.3:53: i/o timeout

After a very long analysis ( I suspected calico for a very long time), I noticed many messages like below in dmesg :

[247338.094016] UDP: bad checksum. From xx.xxx.xxx.xxx:43155 to xx.xxx.xxx.xxx:8472 ulen 75

Long story short, this is a bug in flannel =>flannel-io/flannel#1279

The workaround mentioned in that issue does work ( I applied it on all the nodes as root):

# ethtool -K flannel.1 tx-checksum-ip-generic off
Actual changes:
tx-checksumming: off
       tx-checksum-ip-generic: off
tcp-segmentation-offload: off
       tx-tcp-segmentation: off [requested on]
       tx-tcp-ecn-segmentation: off [requested on]
       tx-tcp6-segmentation: off [requested on]
       tx-tcp-mangleid-segmentation: off [requested on]
udp-fragmentation-offload: off [requested on]

As the fix has been merged only recently, it will be in flannel 0.13.
Until then this workaround is the only solution today.

[floryut]

Hello @HugoRh I feel bad because we identify this and apply this workaround in our test (tests/testcases/040_check-network-adv.yml:19), sorry not to have documented this in the release note...

    - name: Flannel | Disable tx and rx offloading on VXLAN interfaces (see https://github.com/coreos/flannel/pull/1282)
      shell: "ethtool --offload flannel.1 rx off tx off"
      ignore_errors: true
      when:
        - kube_network_plugin|default('calico') == 'flannel'

Anyhow I guess we can close this as it is indeed a flannel bug, nothing to do with kubespray :)

[HugoRh]

Hi @floryut
Well, that warning would have been useful indeed, but shit happens :)
Let's hope next time it reach the release notes.
I raised this issue mostly for documenting this problem which happened during a Kubespray deployment not because of it.
Could you please update the release note to mention that problem?

Thanks !!

[HugoRh]

Also the test might be modified (sorry if the syntax is bad, I begin with Ansible) :

tasks:
    - name: Flannel | Disable tx and rx offloading on VXLAN interfaces (see https://github.com/coreos/flannel/pull/1282)
      shell: "ethtool --offload flannel.1 rx off tx off"
      ignore_errors: true
      when: (kube_network_plugin|default('calico') == 'flannel') or (kube_network_plugin|default('calico') == 'canal')

[floryut]

Also the test might be modified (sorry if the syntax is bad, I begin with Ansible) :

tasks:
    - name: Flannel | Disable tx and rx offloading on VXLAN interfaces (see https://github.com/coreos/flannel/pull/1282)
      shell: "ethtool --offload flannel.1 rx off tx off"
      ignore_errors: true
      when: (kube_network_plugin|default('calico') == 'flannel') or (kube_network_plugin|default('calico') == 'canal')

I would go with something more like kube_network_plugin|default('calico') in ['flannel', 'canal'] but indeed, not sure why our canal tests aren't affected by this

[HugoRh]

I might have a clue here.
to use canal (calico+flannel) you code it kube_network_plugin: canal in ./inventory/<mycluster>/group_vars/k8s-cluster/k8s-cluster.yml
Although canal uses flannel, the test is only looking for flannel explicitly, hence updating the test should cover all cases.

[champtar]

This is a kernel bug, not a flannel bug
kubernetes/kubernetes#88986 (comment)
flannel-io/flannel#1282 (comment)
http://patchwork.ozlabs.org/project/netfilter-devel/patch/335a95d93767f2b58ad89975e4a0b342ee00db91.1587429321.git.gnault@redhat.com/

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[floryut]

/close

[k8s-ci-robot]

@floryut: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.