Collect more info about Amazon VPC CNI

[guessi]

Issue #, if available:

Description of changes:

When troubleshooting with Amazon VPC CNI, it is very depends on how Envs pass into container, if we could get Envs that would be helpful. Besides from these, with detailed info output, we can also tell if aws-node is running with a Service Account, and if IRSA is properly configured, which should benefits a lot for troubleshooting.

Testing Done

Command output with ctr

# timeout 75 ctr --namespace k8s.io container list | awk '/amazon-k8s-cni:v/{print$1}' | xargs -n 1 ctr --namespace k8s.io container info
{
    "ID": "...",
    "Labels": {
        "io.cri-containerd.kind": "container",
        "io.kubernetes.container.name": "aws-node",
        "io.kubernetes.pod.name": "aws-node-n8v84",
        "io.kubernetes.pod.namespace": "kube-system",
        "io.kubernetes.pod.uid": "b30ff7a3-710c-4bf2-b451-0259d5b234cf"
    },
    "Image": "602401143452.dkr.ecr-fips.us-east-1.amazonaws.com/amazon-k8s-cni:v1.11.4",
    "Runtime": {
        "Name": "io.containerd.runc.v2",
        "Options": {
            "type_url": "containerd.runc.v1.Options",
            "value": "SAE="
        }
    },
    "SnapshotKey": "...",
    "Snapshotter": "overlayfs",
    "CreatedAt": "2023-04-27T16:19:30.403535063Z",
    "UpdatedAt": "2023-04-27T16:19:30.403535063Z",
    "Extensions": {
        "io.cri-containerd.container.metadata": {
            "type_url": "github.com/containerd/cri/pkg/store/container/Metadata",
            "value": "..."
        }
    },
    "Spec": {
        "ociVersion": "1.0.2-dev",
        "process": {
            "user": {
                "uid": 0,
                "gid": 0,
                "additionalGids": [
                    0
                ]
            },
            "args": [
                "/app/entrypoint.sh"
            ],
            "env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HOSTNAME=ip-192-168-110-151.ec2.internal",
                "AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER=false",
                "DISABLE_METRICS=false",
                "WARM_ENI_TARGET=1",
                "AWS_STS_REGIONAL_ENDPOINTS=regional",
                "AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token",
                "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=false",
                "AWS_VPC_K8S_CNI_LOG_FILE=/host/var/log/aws-routed-eni/ipamd.log",
                "WARM_PREFIX_TARGET=1",
                "MY_NODE_NAME=ip-192-168-110-151.ec2.internal",
                "ENABLE_IPv4=true",
                "AWS_REGION=us-east-1",
                "AWS_ROLE_ARN=arn:aws:iam::XXXXXX:role/XXX",
                "AWS_VPC_K8S_CNI_RANDOMIZESNAT=prng",
                "AWS_VPC_K8S_CNI_VETHPREFIX=eni",
                "AWS_VPC_K8S_PLUGIN_LOG_FILE=/var/log/aws-routed-eni/plugin.log",
                "AWS_VPC_K8S_PLUGIN_LOG_LEVEL=DEBUG",
                "ADDITIONAL_ENI_TAGS={}",
                "DISABLE_INTROSPECTION=false",
                "DISABLE_NETWORK_RESOURCE_PROVISIONING=false",
                "ENABLE_POD_ENI=false",
                "AWS_VPC_ENI_MTU=9001",
                "AWS_VPC_K8S_CNI_EXTERNALSNAT=false",
                "AWS_VPC_K8S_CNI_LOGLEVEL=DEBUG",
                "ENABLE_PREFIX_DELEGATION=false",
                "AWS_VPC_CNI_NODE_PORT_SUPPORT=true",
                "ENABLE_IPv6=false",
                "AWS_DEFAULT_REGION=us-east-1",
                "AWS_LOAD_BALANCER_WEBHOOK_SERVICE_SERVICE_HOST=10.100.86.249",
                "AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT_443_TCP=tcp://10.100.86.249:443",
                "KUBERNETES_PORT_443_TCP_PROTO=tcp",
                "KUBE_DNS_PORT_53_UDP_ADDR=10.100.0.10",
                "KUBE_DNS_PORT_53_TCP=tcp://10.100.0.10:53",
                "AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT_443_TCP_ADDR=10.100.86.249",
                "KUBERNETES_SERVICE_HOST=10.100.0.1",
                "KUBE_DNS_SERVICE_HOST=10.100.0.10",
                "KUBE_DNS_SERVICE_PORT_DNS_TCP=53",
                "KUBE_DNS_PORT_53_TCP_PROTO=tcp",
                "KUBE_DNS_PORT_53_TCP_ADDR=10.100.0.10",
                "AWS_LOAD_BALANCER_WEBHOOK_SERVICE_SERVICE_PORT=443",
                "AWS_LOAD_BALANCER_WEBHOOK_SERVICE_SERVICE_PORT_WEBHOOK_SERVER=443",
                "KUBERNETES_PORT_443_TCP_PORT=443",
                "KUBE_DNS_PORT_53_UDP_PORT=53",
                "KUBERNETES_PORT_443_TCP=tcp://10.100.0.1:443",
                "KUBERNETES_PORT_443_TCP_ADDR=10.100.0.1",
                "KUBE_DNS_SERVICE_PORT_DNS=53",
                "KUBE_DNS_PORT_53_TCP_PORT=53",
                "AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT_443_TCP_PORT=443",
                "KUBERNETES_PORT=tcp://10.100.0.1:443",
                "KUBE_DNS_PORT=udp://10.100.0.10:53",
                "KUBERNETES_SERVICE_PORT=443",
                "KUBERNETES_SERVICE_PORT_HTTPS=443",
                "AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT=tcp://10.100.86.249:443",
                "KUBE_DNS_SERVICE_PORT=53",
                "KUBE_DNS_PORT_53_UDP=udp://10.100.0.10:53",
                "KUBE_DNS_PORT_53_UDP_PROTO=udp",
                "AWS_LOAD_BALANCER_WEBHOOK_SERVICE_PORT_443_TCP_PROTO=tcp"
            ],
            ...

Command output with docker

# timeout 75 docker container list | awk '/amazon-k8s-cni/{print$NF}' | xargs -n 1 docker container inspect

[
    {
        "Id": "fdc5f8fc29864322384c708a83f6f72080bb095a8230ab4a3763040490aaf222",
        "Created": "2023-06-15T01:12:58.71597003Z",
        "Path": "/app/entrypoint.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 4238,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-06-15T01:12:59.533401526Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        ...
        "HostConfig": {
            "Binds": [
                "/opt/cni/bin:/host/opt/cni/bin",
                "/etc/cni/net.d:/host/etc/cni/net.d",
                "/var/log/aws-routed-eni:/host/var/log/aws-routed-eni",
                "/var/run/aws-node:/var/run/aws-node",
                "/var/run/dockershim.sock:/var/run/dockershim.sock",
                "/run/xtables.lock:/run/xtables.lock",
                "/var/lib/kubelet/pods/7dc2e85a-6acd-4d2a-ad5e-06629a70b556/volumes/kubernetes.io~projected/kube-api-access-tcvtp:/var/run/secrets/kubernetes.io/serviceaccount:ro",
                "/var/lib/kubelet/pods/7dc2e85a-6acd-4d2a-ad5e-06629a70b556/etc-hosts:/etc/hosts",
                "/var/lib/kubelet/pods/7dc2e85a-6acd-4d2a-ad5e-06629a70b556/containers/aws-node/5827febd:/dev/termination-log"
            ],
            ...
        },
        "GraphDriver": ...,
        "Mounts": ...,
        "Config": {
            "Hostname": "ip-192-168-31-214.ec2.internal",
            "Domainname": "",
            "User": "0",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "AWS_VPC_K8S_CNI_VETHPREFIX=eni",
                "DISABLE_METRICS=false",
                "ENABLE_PREFIX_DELEGATION=false",
                "WARM_ENI_TARGET=1",
                "ADDITIONAL_ENI_TAGS={}",
                "AWS_VPC_ENI_MTU=9001",
                "AWS_VPC_K8S_CNI_LOGLEVEL=DEBUG",
                "AWS_VPC_K8S_CNI_RANDOMIZESNAT=prng",
                "DISABLE_NETWORK_RESOURCE_PROVISIONING=false",
                "ENABLE_IPv4=true",
                "ENABLE_IPv6=false",
                "ENABLE_POD_ENI=false",
                "AWS_VPC_CNI_NODE_PORT_SUPPORT=true",
                "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=false",
                "AWS_VPC_K8S_CNI_EXTERNALSNAT=false",
                "AWS_VPC_K8S_CNI_LOG_FILE=/host/var/log/aws-routed-eni/ipamd.log",
                "AWS_VPC_K8S_PLUGIN_LOG_FILE=/var/log/aws-routed-eni/plugin.log",
                "AWS_VPC_K8S_PLUGIN_LOG_LEVEL=DEBUG",
                "DISABLE_INTROSPECTION=false",
                "MY_NODE_NAME=ip-192-168-31-214.ec2.internal",
                "AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER=false",
                "WARM_PREFIX_TARGET=1",
                "KUBERNETES_SERVICE_HOST=10.100.0.1",
                "KUBERNETES_PORT_443_TCP_PORT=443",
                "KUBE_DNS_SERVICE_HOST=10.100.0.10",
                "KUBE_DNS_SERVICE_PORT=53",
                "KUBE_DNS_PORT_53_UDP_PROTO=udp",
                "KUBE_DNS_PORT_53_UDP_ADDR=10.100.0.10",
                "KUBE_DNS_PORT_53_TCP_PORT=53",
                "KUBE_DNS_PORT_53_TCP_ADDR=10.100.0.10",
                "KUBERNETES_SERVICE_PORT=443",
                "KUBERNETES_SERVICE_PORT_HTTPS=443",
                "KUBERNETES_PORT_443_TCP_ADDR=10.100.0.1",
                "KUBE_DNS_SERVICE_PORT_DNS=53",
                "KUBE_DNS_SERVICE_PORT_DNS_TCP=53",
                "KUBE_DNS_PORT=udp://10.100.0.10:53",
                "KUBE_DNS_PORT_53_TCP_PROTO=tcp",
                "KUBE_DNS_PORT_53_UDP=udp://10.100.0.10:53",
                "KUBERNETES_PORT=tcp://10.100.0.1:443",
                "KUBERNETES_PORT_443_TCP=tcp://10.100.0.1:443",
                "KUBERNETES_PORT_443_TCP_PROTO=tcp",
                "KUBE_DNS_PORT_53_UDP_PORT=53",
                "KUBE_DNS_PORT_53_TCP=tcp://10.100.0.10:53",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "NONE"
                ]
            },
            "Image": "602401143452.dkr.ecr.us-east-1.amazonaws.com/amazon-k8s-cni@sha256:19dacc4b46485c85ddfbee3545d34948bad08ce318c0fa997541546786aa7bc4",
            "Volumes": null,
            "WorkingDir": "/app",
            "Entrypoint": [
                "/app/entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": ...
        },
        "NetworkSettings": ...
    }
]

[guessi]

@cartermckinnon Could you please spare some time to review this change, thanks!

[cartermckinnon]

What does the output from Docker look like?

[cartermckinnon]

I assume the Timed out case is what happens when there is no CNI container running?

[guessi]

If there is no CNI running, awk and xargs magic should not work and the output should be empty.

There should have no timeout case here? To me, it's more likely to be a safe check to ensure the script will eventually be ended.

[guessi]

docker would have similar output, I will post here later.

[guessi]

Just updated command output for docker