Recurrent error in zonal controlplane: TLS handshake error from xxx.xxx.xxx.xxx:yyy: EOF

[slavogiez]

What happened?

In our zonal controlplane deployed on GKE, we can see this error appearing again and again:

kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:25:11 http: TLS handshake error from 172.17.1.17:44366: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:28:11 http: TLS handshake error from 172.17.4.239:52698: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:29:11 http: TLS handshake error from 172.17.0.7:49890: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:30:11 http: TLS handshake error from 172.17.4.239:38510: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:31:11 http: TLS handshake error from 172.17.0.7:38410: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:32:16 http: TLS handshake error from 172.17.4.239:48328: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:35:11 http: TLS handshake error from 172.17.5.31:34122: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:39:11 http: TLS handshake error from 172.17.5.31:53686: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:39:11 http: TLS handshake error from 172.17.5.31:53698: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:40:11 http: TLS handshake error from 172.17.4.239:53264: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:42:17 http: TLS handshake error from 172.17.4.55:48322: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:43:11 http: TLS handshake error from 172.17.4.55:44964: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:44:11 http: TLS handshake error from 172.17.4.239:44704: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:46:11 http: TLS handshake error from 172.17.0.7:60780: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:49:11 http: TLS handshake error from 172.17.4.55:60228: EOF
kong-mesh-control-plane-5584596d6f-2pm78 2023/11/03 09:50:11 http: TLS handshake error from 172.17.4.239:44684: EOF
kong-mesh-control-plane-5584596d6f-2pm78 2023/11/03 09:51:11 http: TLS handshake error from 172.17.4.239:59148: EOF
kong-mesh-control-plane-5584596d6f-6tc4m 2023/11/03 09:51:11 http: TLS handshake error from 172.17.4.239:52766: EOF

We figured out that the source IPs where from the konnectivity-agent pods from kube-system namespace.
GKE version: v1.27.3-gke.100

After doing some searches, it seems that this error is related to a bug in Kubernetes/Golang.

Can you confirm that? Is there a way to prevent this error?
Thanks!

[slavogiez]

This looks like the same issue as :

• TLS handshake error: EOF cert-manager/cert-manager#4594
• TLS handshake error from: EOF open-policy-agent/gatekeeper#2142

[michaelbeaumont]

Just a note, this is easy to reproduce with a default cluster on said GKE version and default Kuma zone CP install

[lahabana]

Triage: looks like a bug in the konnectivity agent. Let's reproduce and confirm this. If it is a bug in Konnectivity let's xref this to the issue there and close this.

[Automaat]

I've managed to reproduce it easily on GKE v1.27.3-gke.100, this looks like konnectivity-agent issue, and we are unable to do anything about it