Load balancer node unhealthy

[ludgart]

Hello,

I'm trying to set up a k3s. In my following test environment, have the following servers:

• master1
• worker1

Both servers run with Debian 11 and with the k3s version v1.21.5+k3s1.
For my network infrastructure, I'm using cilium.

The problem is, on my load balancer the master node show healthy, but the worker node unhealthy.

Hetzner network: 10.0.0.0/16
Cluster CIDR: 10.244.0.0/16

For the cilium/network installation,
I followed https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/docs/deploy_with_networks.md with the cilium.yml values https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/e2etests/templates/cilium.yml

kubectl get nodes -A

NAME      STATUS   ROLES                       AGE   VERSION
master1   Ready    control-plane,etcd,master   23h   v1.21.5+k3s1
worker1   Ready    <none>                      23h   v1.21.5+k3s1

kubectl get pods -A -o wide

NAMESPACE     NAME                                               READY   STATUS      RESTARTS   AGE     IP             NODE      NOMINATED NODE   READINESS GATES
default       webtest1-57fcf49d9b-qgfp7                          1/1     Running     0          9m57s   10.244.1.82    worker1   <none>           <none>
kube-system   cilium-64qlq                                       1/1     Running     1          23h     10.0.0.3       worker1   <none>           <none>
kube-system   cilium-mfbsn                                       1/1     Running     1          23h     10.0.0.2       master1   <none>           <none>
kube-system   cilium-operator-5d5fd9d85d-6frcb                   1/1     Running     1          23h     10.0.0.2       master1   <none>           <none>
kube-system   cilium-operator-5d5fd9d85d-dpsqp                   1/1     Running     2          23h     10.0.0.3       worker1   <none>           <none>
kube-system   coredns-7448499f4d-vhmmn                           1/1     Running     1          23h     10.244.0.130   master1   <none>           <none>
kube-system   hcloud-cloud-controller-manager-666d7bbfcc-fgt84   1/1     Running     1          23h     10.0.0.2       master1   <none>           <none>
kube-system   helm-install-traefik-crd-2qjc9                     0/1     Completed   0          23h     10.244.0.243   master1   <none>           <none>
kube-system   helm-install-traefik-pthk6                         0/1     Completed   1          23h     10.244.0.170   master1   <none>           <none>
kube-system   metrics-server-86cbb8457f-thx8v                    1/1     Running     1          23h     10.244.0.144   master1   <none>           <none>
kube-system   svclb-traefik-79vz5                                2/2     Running     2          23h     10.244.1.222   worker1   <none>           <none>
kube-system   svclb-traefik-z4l2l                                2/2     Running     2          23h     10.244.0.214   master1   <none>           <none>
kube-system   traefik-97b44b794-tn6fw                            1/1     Running     1          23h     10.244.0.134   master1   <none>           <none>

Traefik service

apiVersion: v1
kind: Service
metadata:
  annotations:
    load-balancer.hetzner.cloud/location: nbg1
    load-balancer.hetzner.cloud/use-private-ip: "true"
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2022-02-02T18:57:58Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-9.18.2
  name: traefik
  namespace: kube-system
  resourceVersion: "192706"
  uid: aece1564-1449-4518-9c22-2d393322233b
spec:
  clusterIP: 10.43.25.240
  clusterIPs:
  - 10.43.25.240
  externalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web
    nodePort: 31153
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 30266
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 78.46.xxx.xxx
    - ip: 88.198.xxx.xxx

kubectl get events -A

NAMESPACE     LAST SEEN   TYPE      REASON                   OBJECT                           MESSAGE
kube-system   14m         Normal    EnsuringLoadBalancer     service/traefik                  Ensuring load balancer
kube-system   14m         Normal    EnsuredLoadBalancer      service/traefik                  Ensured load balancer

kubectl logs svc/traefik -n kube-system

time="2022-02-02T19:53:01Z" level=info msg="Configuration loaded from flags."
W0202 19:53:02.561979       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0202 19:53:02.587312       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0202 19:53:02.998332       1 warnings.go:70] networking.k8s.io/v1beta1 IngressClass is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 IngressClassList

I think I'm running into the same issues as #212, but I already tried it without success.
I thank you in advance!

Thanks in advance
ludgart

[dclayton77]

I had a very similar issue using Kong Ingress. I had to install Kong as a daemonset so that the load balancer could check the health of each node. Its not the same setup as you but I have written a blog post series about getting a full cluster setup and running.

https://dcse-ltd.co.uk/building-a-bare-metal-kubernetes-cluster-part-3-service-mesh-https-certificates-and-ingress/

[emrahcetiner]

@ludgart you can try externalTrafficPolicy with Local value

externalTrafficPolicy: Local

[github-actions]

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.

[ludgart]

The problem still exists.