Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Difficult to filter netpol errors #7538

Closed
manuelbuil opened this issue May 12, 2023 · 1 comment
Closed

Difficult to filter netpol errors #7538

manuelbuil opened this issue May 12, 2023 · 1 comment
Assignees
Milestone

Comments

@manuelbuil
Copy link
Contributor

manuelbuil commented May 12, 2023

Environmental Info:
K3s Version:

Any

Node(s) CPU architecture, OS, and Version:

Cluster Configuration:

Describe the bug:

When the network policy controller fails to start, the error is thrown on the logs but it is impossible to know if the error is coming from the network policy controller

Steps To Reproduce:

  • Installed K3s:
    When deploying k3s in dualStack mode using an interface different from the default one, you get the error. For example, we can do that with tailscale:

1 - Install tailscale in the node: curl -fsSL https://tailscale.com/install.sh | sh. That will create a tailscale0 interface
2 - execute tailscale up and login. That should give tailscale0 two IP addresses, one IPv4 and one IPv6
3 - Deploy k3s with the following config:

write-kubeconfig-mode: 644
token: "secret"
cluster-cidr: 10.42.0.0/16,2001:cafe:42:0::/56
service-cidr: 10.43.0.0/16,2001:cafe:42:1::/112
node-ip: "$tailscale0_IPV4,$tailscale0_IPV6"

Replace $tailscale0_IPV4 and $tailscale0_IPV6 by the ip addresses of the tailscale0 interface
4 - After ~20 seconds, k3s installation will fail. You will see in the journalctl logs:
level=fatal msg="IPv6 was enabled but no IPv6 address was found on node"

Expected behavior:

Errors wrapped with some sort of string referring to network policy controller

Actual behavior:

raw errors without any context

Additional context / logs:

@VestigeJ
Copy link

##Environment Details
VERSION=v1.27.1+k3s1
COMMIT=cdcd4a900030416da21701fc0d47d8f4977d5d21

Infrastructure

  • Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150400.24.11-default x86_64 GNU/Linux 
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP4"

Cluster Configuration:

1 server

Config.yaml:

write-kubeconfig-mode: 644
debug: true
token: verymanyoddbatteries
cluster-init: true
node-ip: TAILSCALE_IPV4,TAILSCALE_IPV6
cluster-cidr: 10.42.0.0/16,2001:cafe:42:0::/56
service-cidr: 10.43.0.0/16,2001:cafe:42:1::/112

Existing journalctl log data

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ curl -fsSL https://tailscale.com/install.sh | sh
$ sudo tailscale up
$ ip a //get tailscale ipv4/6
$ sudo INSTALL_K3S_VERSION=v1.27.1+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh 
$ sudo journalctl -u k3s 
$ get_report //generate this template

Results:

$ sudo journalctl -u k3s output on existing release v1.27.1+k3s1

May 18 04:00:37 ip-1-1-2-27 k3s[4934]: time="2023-05-18T04:00:37Z" level=fatal msg="IPv6 was enabled but no IPv6 address was found on node"

Confirmed more informative logging output

Validation Steps

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ curl -fsSL https://tailscale.com/install.sh | sh
$ sudo tailscale up
$ ip a //get tailscale ipv4/6
$ sudo INSTALL_K3S_COMMIT=cdcd4a900030416da21701fc0d47d8f4977d5d21 INSTALL_K3S_EXEC=server ./install-k3s.sh 
$ sudo journalctl -u k3s 
$ get_report //generate this template

Results:

on latest commit_id cdcd4a9

 k3s[4745]: time="2023-05-18T04:01:36Z" level=fatal msg="unable to initialize Network Policy Controller: IPv6 was enabled but no IPv6 address was found on node"
May 18 04:01:36 ip-172-31-27-225 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

3 participants