Formally add support for CentOS 7

[davidnuzik]

We need to expand our testing and identify any issues that prevent us from formally supporting CentOS. Keep in mind K3s is expected to work fine on CentOS 7. This issue is to track the testing effort required to formally support and certify the operating system (See https://rancher.com/docs/k3s/latest/en/installation/node-requirements/#operating-systems )

Currently there are existing issues with the os/centos label, but take care to note that these issues are not all necessarily caused just by utilizing CentOS. As such, it makes sense to review those GitHub issues, but we need to execute some testing and identify any other issues. As needed, we'll need to resolve these issues so we may fully support CentOS.

SELinux support is also needed, which is tracked separately here: #1372

gz#9311

gz#9743

[davidnuzik]

@ShylajaDevadiga I have assigned this issue to you for now. This will require some testing and discovery. We need to identify any/all CentOS issues that prevent us from formally supporting CentOS in our next release. Work with me as needed.

[davidnuzik]

As a reminder we must support IPv6 as well.

[ThomasADavis]

iptables in Centos8 is now legacy.. they now use iptables-nft.

so on a centos 8 system, using iptables gives you this:

[root@mouse-r13 ~]# iptables -L -v -n
Chain INPUT (policy ACCEPT 154K packets, 264M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 957 packets, 53559 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 58311 packets, 10M bytes)
 pkts bytes target     prot opt in     out     source               destination         
# Warning: iptables-legacy tables present, use iptables-legacy to see them
[root@mouse-r13 ~]# iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 990 packets, 73523 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain INPUT (policy ACCEPT 59 packets, 3564 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 693 packets, 45283 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 620 packets, 39433 bytes)
 pkts bytes target     prot opt in     out     source               destination         
# Warning: iptables-legacy tables present, use iptables-legacy to see them

and I believe firewalld is also not supported by k3s.

This doesn't mean k3s does not work, it's just not possible to see the iptables rules.

[Lohann]

+1

[Lohann]

Related #401 #1019

[philipsparrow]

I have been documenting all the steps I needed to get it working in CentOS7, I'll gladly share those steps. It worked out of the box on a Google Cloud VM but not on a local, freshly installed instance.
Namely, installation of iptables and removal of firewalld and wiping out reject rules from INPUT and FORWARD chains and installation of semanage. I'll gladly share these steps although my procedure is a little heavy-handed

[ThomasADavis]

Reader digest version: Don't use Centos v8 because of nft/legacy iptables problems.

So, to help clarify - in my reasearch, RHEL/Centos8 uses nft for iptables, not iptables. At this time, nft is not supported by Kubernetes. There is iptables/iptables-legacy support, and what will happen is the rules are still created and executed, but in RHEL/Centos8, they do not live in harmony with any other nft/iptables, unless it's the only ruleset you want to run.

You cannot see these iptables rule sets by default, since they occur in the legacy iptables rules due to the container having it's own iptables (not nft the binaries), and RHEL/Centos8 does not provide the legacy iptable tools.

There are other distributions heading towards using nft instead of iptables, but so far, it appears that they do include the legacy iptable binaries.

This means until nft is in Kubernetes (not k3s), RHEL/Centos8 and other distribution using nft tables is not truly supported.

[ThomasADavis]

Well, maybe it's not so bad for RHEL/Centos8..

see kubernetes/kubernetes#71305

[sraillard]

@philipsparrow I would be interessed in the steps needed to make k3s works on CentOS 7.7. Even after a fresh install, removing firewalld, disabling SELinux, installing iptables-service, adding "user_namespace.enable=1" to the kernel command line, k3s is still not vworking... It's looking like a network issue as the API server isn't reachable.

[Lohann]

@sraillard I wrote a step-by-step here, let me know if it works for you:
#1019 (comment)

[philipsparrow]

I don't think I have anything as good as @Lohann has provided, I got it working with only the following steps (caveat: I don't need Traefik so haven't worked on that):

systemctl stop firewalld
systemctl disable firewalld
yum update
yum install -y iptables-services policycoreutils-python
systemctl start iptables
systemctl enable iptables
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
iptables -F
# This gets rid of any DROP rules in the INPUT and FORWARD chains
iptables-save > /etc/sysconfig/iptables
reboot now

Then I installed K3s with no special options.
FYI in my debugging, I found it enormously helpful to check both routes and firewall. Sometimes I was missing routes. ip a and ip route are your friend. From my cluster (single master, 2 worker nodes, flannel VXLAN) I expect to see routes that look like:

default via 10.126.126.1 dev eth0 proto dhcp metric 100
10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink
10.42.2.0/24 via 10.42.2.0 dev flannel.1 onlink
10.126.126.1 dev eth0 proto dhcp scope link metric 100
10.126.126.3 dev eth0 proto kernel scope link src 10.126.126.3 metric 100

I hope this helps

[brandond]

Yeah to me "a system where SELinux is enabled by default" means enforcing or permissive - not absent or disabled. Maybe worth a clarifying change to the docs?

[noelmcloughlin]

I remember the issue now. Running the script failed. It did not say you should have selinux=enforcing or set INSTALL_K3S_SELINUX_WARN=true but instead threw an error message saying "ensure selinux-policy-base is installed" so that indicated a packaging problem, not a SELinux != enforcing issue. The script error confused me.

[cjellick]

@ShylajaDevadiga I dont think I want this issue closed until CentOS 7 is 100% validated. I can't see that happening until the conformance tests pass cleanly and successfully on an officially release. So, I htink its fine that you opened an issues specifically for the conformance test failures, but this issue should be held open until that one works.

[cjellick]

To be honest, I'm also not sure that we can claim cent 7 support without revisiting selinux.

[ShylajaDevadiga]

Closing issue as conformance tests have passed.Results tracked in #1960.

[bbhenry]

I just recently deployed K3S to a CentOS7 server. K3S was installed but the pods were not able to communicate to the api server just like described before. I had to disable firewalld to get things working. How is this ticket closed if the latest K3S should work on a CentOS7 environment? Am I missing something?

[Fodoj]

IMO this is expected behavior if you have firewall enabled. Installation of K3s doesn’t handle complete server configuration (correct me if I am wrong).

[brandond]

That is correct. It works on RHEL7 if you don't break it by blocking traffic or doing other things that would prevent it from working.

[sraillard]

I agree that k3s can't configure all the server settings. The fact is that firewalld is by default enabled, so that's classic issue (and many people have it). Maybe a solution could be checking some firewall rules and printing a warning if some rules may prevent k3s from working correctly?

[Fodoj]

For example, CentOS 7 AMIs (and I guess other cloud images) have firewalld disabled by default, but yeah, standard ISO installation has it enabled normally. But then, firewall could be also outside the server and also break K8s/K3s.

[WillPlatnick]

Is there anything blocking calling CentOS officially supported now?

[akhfa]

Is there anything blocking calling CentOS officially supported now?

I'm also waiting for this. Can we add Centos to the operating system requirements in the docs?

[bluebrown]

I am getting this on centos8

I haven't found a way to use legacy IP tables on CentOS yet. I think it may not be possible. I could still run the dashboard though.

[noelmcloughlin]

On saltstack-formulas we run CI/CD that includes installing K3S on various OS (https://travis-ci.com/github/saltstack-formulas/kubernetes-formula) using binary (https://github.com/rancher/k3s/releases).

                 ID: kubernetes-k3s-binary-prerequisites
           Function: file.directory
               Name: /usr/local/k8s-k3s-1.18.4+k3s1/bin
             Result: True
            Comment: Directory /usr/local/k8s-k3s-1.18.4+k3s1/bin updated
            Started: 22:41:32.870731
           Duration: 3.539 ms
            Changes:   
              ----------
              /usr/local/k8s-k3s-1.18.4+k3s1/bin:
                  New Dir
       ----------
                 ID: kubernetes-k3s-binary-install
           Function: file.managed
               Name: /usr/local/k8s-k3s-1.18.4+k3s1/bin/k3s
             Result: True
            Comment: File /usr/local/k8s-k3s-1.18.4+k3s1/bin/k3s updated
            Started: 22:41:32.876005
           Duration: 1625.91 ms
            Changes:   
              ----------
              diff:
                  New file
              mode:
                  0755
       ----------
                 ID: kubernetes-k3s-binary-install-symlink
           Function: file.symlink
               Name: /usr/local/bin/k3s
             Result: True
            Comment: Created new symlink /usr/local/bin/k3s -> /usr/local/k8s-k3s-1.18.4+k3s1/bin/k3s
            Started: 22:41:34.504775
           Duration: 70.67 ms
            Changes:   
              ----------
              new:
                  /usr/local/bin/k3s

I would like to add job to run the K3S script (https://get.k3s.io) on CentOS7 and CentOS8 but the hash sum is missing.

K3S should provide a hash sum (i.e. https://get.k3sum.io) so downloaded script can be verified.

That would be a nice improvement.

[brandond]

@noelmcloughlin can you create a new feature request issue for that? I don't think it's related to this issue.

[kylos101]

For posterity, with CentOS7.8, I found I had to make a slight change to update kernel (notice the second sudo for the --update-kernel argument:

sudo grubby --args="user_namespace.enable=1" --update-kernel="$(sudo grubby --default-kernel)"    

[PRNDA]

For posterity, with CentOS7.8, I found I had to make a slight change to update kernel (notice the second sudo for the --update-kernel argument:

sudo grubby --args="user_namespace.enable=1" --update-kernel="$(sudo grubby --default-kernel)"    

what's it for? and why this?

[kylos101]

@PRNDA

To enable user namespaces in centos/7. The k3s check-config was failing with ootb with centos/7