Pod communication problems across nodes

[itechbear]

When I set up a kubernetes cluster, using flannel as the CNI plugin, I have trouble to access pods on different nodes. Pods on the same node have no trouble accessing each other (DNS queries, curl Web services, etc).

Expected Behavior

Pods across nodes should have no trouble accessing each other.

Context

I have a kubernetes cluster with two nodes. I installed both docker and containerd on each node, for some historical reasons. I use docker to run some containers and use containerd as the kubernetes runtime. When I set up the cluster using kubeadm and flannel as the CNI plugin, I found that pods on different nodes have troubles accessing each other, while those on the same node work normally.

What I observed so far:

1. The flannel port is open and seems working when I use nc -vz -u HOST 8472 to check it.
2. Pods on different nodes can ping each other.
3. Traceroute to pod on other nodes with ICMP/TCP SYN would work, while it would fail using the default UDP protocol.

What I have tried: reset the cluster and then set up the kubernetes cluster.
What I haven't tried: remove the docker daemon from the nodes or restart the nodes.

Your Environment

• Flannel version: 0.24.4
• Backend used (e.g. vxlan or udp): vxlan (All config default)
• Etcd version: 3.5.7-0
• Kubernetes version (if used): 1.27.2
• Operating System and version: CentOS 7.3 3.10.0-1160.7.1
• Link to your project (optional):

[itechbear]

UPDATE:
Just got the solution as mentioned in #1929

On each node, execute the following command:sudo ethtool -K flannel.1 tx-checksum-ip-generic off