Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.28] - Containerd CRI registry mirrors and configs config via config.toml have been deprecated #9131

Closed
brandond opened this issue Jan 2, 2024 · 1 comment
Closed

[Release-1.28] - Containerd CRI registry mirrors and configs config via config.toml have been deprecated #9131

brandond opened this issue Jan 2, 2024 · 1 comment
Assignees
@VestigeJ
Milestone
v1.28.6+k3s1

Comments

@brandond
Copy link
Member

brandond commented Jan 2, 2024

Backport fix for Containerd CRI registry mirrors and configs config via config.toml have been deprecated
@caroline-suse-rancher caroline-suse-rancher added this to the v1.28.6+k3s1 milestone Jan 3, 2024
@brandond brandond mentioned this issue Jan 17, 2024
Closed
@VestigeJ VestigeJ self-assigned this Jan 22, 2024
@VestigeJ
Copy link

##Environment Details
Validated using VERSION=v1.28.6-rc2+k3s1

Infrastructure

  • Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.55.44-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME              STATUS   ROLES                       AGE   VERSION
ip-17-13-12-167   Ready    control-plane,etcd,master   48s   v1.28.6-rc2+k3s1

Config.yaml:

token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
protect-kernel-defaults: true
cluster-init: true

Validation

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

 version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/41baf3230c69f60cbc3c461f264a8a80892b78b4841a8dbe43d2feb6e297c541/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic"]
  runtime_type = "io.containerd.lunatic.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic".options]
  BinaryName = "/usr/local/bin/containerd-shim-lunatic-v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/local/bin/nvidia-container-runtime"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-experimental"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-experimental".options]
  BinaryName = "/usr/local/bin/nvidia-container-runtime-experimental"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"]
  runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options]
  BinaryName = "/usr/local/bin/containerd-shim-spin-v1"

$ sudo INSTALL_K3S_VERSION=v1.28.6-rc2+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh

[INFO]  Using v1.28.6-rc2+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.29.1-rc2+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.29.1-rc2+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

$ set_kubefig //export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

$ kg runtimeclass

kg runtimeclass
NAME                  HANDLER               AGE
crun                  crun                  83s
lunatic               lunatic               83s
nvidia                nvidia                83s
nvidia-experimental   nvidia-experimental   83s
slight                slight                83s
spin                  spin                  83s
wasmedge              wasmedge              83s
wasmer                wasmer                83s
wasmtime              wasmtime              83s
wws                   wws                   83s

get_wasi() {
    has_bin wget
    _arch=$(uname -m)
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v1-lunatic-linux-"${_arch}".tar.gz
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v1-slight-linux-"${_arch}".tar.gz
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v1-wws-linux-"${_arch}".tar.gz
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v2-spin-linux-"${_arch}".tar.gz
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/workload.yaml
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/runtime.yaml

    tar -xvf containerd-wasm-shims-v1-lunatic-linux-"${_arch}".tar.gz
    tar -xvf containerd-wasm-shims-v1-slight-linux-"${_arch}".tar.gz
    tar -xvf containerd-wasm-shims-v1-wws-linux-"${_arch}".tar.gz
    tar -xvf containerd-wasm-shims-v2-spin-linux-"${_arch}".tar.gz

    chmod +x containerd-shim-lunatic-v1
    chmod +x containerd-shim-slight-v1
    chmod +x containerd-shim-wws-v1
    chmod +x containerd-shim-spin-v2

    sudo cp containerd-shim-lunatic-v1 /usr/local/bin/
    sudo cp containerd-shim-slight-v1 /usr/local/bin/
    sudo cp containerd-shim-wws-v1 /usr/local/bin/
    sudo cp containerd-shim-spin-v2 /usr/local/bin/
    # call host-config function to write config.toml file
    set_runtimes
}

set_runtimes() {
  cat <<EOF > config.toml.tmpl
  # File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/41baf3230c69f60cbc3c461f264a8a80892b78b4841a8dbe43d2feb6e297c541/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic"]
  runtime_type = "io.containerd.lunatic.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic".options]
  BinaryName = "/usr/local/bin/containerd-shim-lunatic-v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/local/bin/nvidia-container-runtime"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-experimental"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-experimental".options]
  BinaryName = "/usr/local/bin/nvidia-container-runtime-experimental"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"]
  runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options]
  BinaryName = "/usr/local/bin/containerd-shim-spin-v1"

EOF
sudo mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/
sudo cp config.toml.tmpl /var/lib/rancher/k3s/agent/etc/containerd/
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VestigeJ VestigeJ

Labels
None yet
Projects
K3s Development
Status: Done Issue
Milestone
v1.28.6+k3s1
Development

No branches or pull requests
3 participants
@brandond @VestigeJ @caroline-suse-rancher