The ImagePullPolicy was set but it didn't seem to work

[mutezebra]

I have a pod, I use Dockerfile to build his image and package it as a tar file, then use the ctr command to extract the tar file into the container, then I use the kubectl command to try to run my pod, but the pod fails to run, showing that he did not find the image. However, I used the ctr images list command to find that there is actually a mirror, and my yaml file does define ImagePullPolicy as IfNotPresent,and if I set imagePullPolicy to Never, he still can't find the image。

1. kubectl get pod

mz@ubuntu:~/tiktok$ kub get pod
NAME                                      READY   STATUS             RESTARTS   AGE
tiktok-user-dep-7f9d5c469d-5gkpc          0/1     ImagePullBackOff   0          6m28s

2. kubctl describe pod

Name:             tiktok-user-dep-7f9d5c469d-5gkpc
Namespace:        default
Priority:         0
Service Account:  default
Node:             ubuntu/192.168.84.6
Start Time:       Tue, 02 Jul 2024 01:44:52 -0700
Labels:           app=tiktok-user-pod
                  pod-template-hash=7f9d5c469d
Annotations:      <none>
Status:           Pending
IP:               10.42.0.29
IPs:
  IP:           10.42.0.29
Controlled By:  ReplicaSet/tiktok-user-dep-7f9d5c469d
Containers:
  user:
    Container ID:
    Image:          mutezebra/tiktok-user:2.0
    Image ID:
    Port:           10001/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /app/config/config.yaml from config (rw,path="config.yaml")
      /app/logs from logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7jx6q (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      tiktok-user-config
    Optional:  false
  logs:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  tiktok-user-log-pvc
    ReadOnly:   false
  kube-api-access-7jx6q:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  8m5s                  default-scheduler  Successfully assigned default/tiktok-user-dep-7f9d5c469d-5gkpc to ubuntu
  Normal   Pulling    6m42s (x4 over 8m5s)  kubelet            Pulling image "mutezebra/tiktok-user:2.0"
  Warning  Failed     6m40s (x4 over 8m2s)  kubelet            Failed to pull image "mutezebra/tiktok-user:2.0": failed to pull and unpack image "docker.io/mutezebra/tiktok-user:2.0": failed to resolve reference "docker.io/mutezebra/tiktok-user:2.0": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     6m40s (x4 over 8m2s)  kubelet            Error: ErrImagePull
  Warning  Failed     6m16s (x6 over 8m2s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    3m (x20 over 8m2s)    kubelet            Back-off pulling image "mutezebra/tiktok-user:2.0"

3. ctr -n k8s.io images list

mz@ubuntu:~/20/2024Summer/tiktok$ sudo ctr -n k8s.io images list
REF                                               TYPE                                       DIGEST
                                             SIZE      PLATFORMS   LABELS
docker.io/mutezebra/tiktok-user:2.0               application/vnd.oci.image.manifest.v1+json sha256:77b664615cae3b8f2e23f8629c29f9ed676e53bf0923ad6bfe1e453a7ffe7e8d 48.3 MiB  linux/amd64 -
docker.io/rancher/klipper-lb:v0.4.7               application/vnd.oci.image.manifest.v1+json sha256:c27be397ca12f7758420681a5c47d0f2ab907276824a4bc4bcb6c44c369d9070 11.7 MiB  linux/amd64 -
docker.io/rancher/mirrored-coredns-coredns:1.10.1 application/vnd.oci.image.manifest.v1+json sha256:e42d91b3fdc8a8c2b527a5e3144f47cadbb768669b3f6f1fd22b4d46b0bc2526 51.1 MiB  linux/amd64 -
docker.io/rancher/mirrored-library-traefik:2.10.7 application/vnd.oci.image.manifest.v1+json sha256:bf7f15a4ad5da03ff5e4285d33d8e8eeeccd6f3bc60b63d83edf9fc809351d5d 146.8 MiB linux/amd64 -

4. yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tiktok-user-dep
spec:
  selector:
    matchLabels:
      app: tiktok-user-pod
  replicas: 1
  template:
    metadata:
      labels:
        app: tiktok-user-pod
    spec:
      containers:
        - name: user
          image: mutezebra/tiktok-user:2.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 10001
          volumeMounts:
            - name: config
              mountPath: /app/config/config.yaml
              subPath: config.yaml
            - name: logs
              mountPath: /app/logs

      volumes:
        - name: config
          configMap:
            name: tiktok-user-config
            items:
              - key: config.yaml
                path: config.yaml
        - name: logs
          persistentVolumeClaim:
            claimName: tiktok-user-log-pvc

5. Set imgaePullPolicy to Never

mz@ubuntu:~/tiktok$ kub get pod
NAME                                      READY   STATUS             RESTARTS   AGE
tiktok-user-dep-7f9d5c469d-hxnff          0/1     ErrImageNeverPull 0          1m

6. kubectl get pod -n kube-system

mz@ubuntu:~/20/2024Summer/tiktok$ kub get pod -n kube-system
NAME                                     READY   STATUS      RESTARTS   AGE
coredns-6799fbcd5-4jz6s                  1/1     Running     0          10h
helm-install-traefik-crd-b9mhb           0/1     Completed   0          10h
helm-install-traefik-pvlsg               0/1     Completed   2          10h
local-path-provisioner-6f5d79df6-nr2zq   1/1     Running     0          10h
metrics-server-54fd9b65b-qgqpb           1/1     Running     0          10h
svclb-traefik-5cf2f30d-2h7cf             2/2     Running     0          10h
traefik-7d5f6474df-twcqq                 1/1     Running     0          10h

[mutezebra]

Its the result of k3s check-config

mz@ubuntu:~/tiktok$ sudo k3s check-config
Verifying binaries in /var/lib/rancher/k3s/data/a3c3f3dba66f03ed65b7c1266e2afdb9c108e60efd20c57fa804a3931f5febad/bin:
- sha256sum: good
- links: good

System:
- /usr/sbin iptables v1.8.4 (legacy): ok
- swap: should be disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-113-generic
info: reading kernel config from /boot/config-5.15.0-113-generic ...

Generally Necessary:
- cgroup hierarchy: cgroups Hybrid mounted, cpuset|memory controllers status: good
- /usr/sbin/apparmor_parser
apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_IP_NF_TARGET_REJECT: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_SET: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled (as module)
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: missing
- Storage Drivers:
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)

STATUS: pass

[brandond]

then use the ctr command to extract the tar file into the container, then I use the kubectl command to try to run my pod, but the pod fails to run, showing that he did not find the image.

I'm not following here. Why are you extracting the tar file into the container. Are you running k3s and containerd in a container? Is this a single-node cluster? Are you sure that the ctr -n k8s.io images list output is from the same containerd instance that the kubelet is talking to - are they on the same node, and neither of them running IN a container?

[mutezebra]

Are you sure that the ctr -n k8s.io images list output is from the same containerd instance

You are right. It not the same containerd instance. I remove the containerd from my host, then it works. Thank you very much. It puzzled me for a long time。