Warmer behaviour changed in the 1.12.0 release breaking pipelines

[RoSk0]

We are using gcr.io/kaniko-project/executor:debug to get latest features of the Kaniko so spotted this quite quickly.

To avoid building image for the same commit twice, we use this technique:

if warmer --image $CI_REGISTRY_IMAGE/$BUILD_NAME:$CI_COMMIT_SHORT_SHA; then echo "No need to build image - it exists already!"; exit 0; fi

Actual behavior

The output is full, but redacted:

Executing "step_script" stage of the job script 00:17
$ if warmer --image $CI_REGISTRY_IMAGE/$BUILD_NAME:$CI_COMMIT_SHORT_SHA; then echo "No need to build image - it exists already!"; exit 0; fi
INFO[0000] Retrieving image manifest gitlab.server:4567/php:523e6a58
INFO[0000] Retrieving image gitlab.server:4567/php:523e6a58 from registry gitlab.server:4567
WARN[0000] Error while trying to warm image: gitlab.server:4567/php:523e6a58 Failed to retrieve image: gitlab.server:4567/php:523e6a58: GET https://gitlab.server:4567/v2/php/manifests/523e6a58: MANIFEST_UNKNOWN: manifest unknown; map[Tag:523e6a58]
WARN[0000] Error while trying to warm image: : Failed to verify image name: :: could not parse reference: :
INFO[0000] Retrieving image manifest docker.io/bitnami/memcached:1
INFO[0000] Retrieving image docker.io/bitnami/memcached:1 from registry index.docker.io
INFO[0005] Retrieving image manifest docker.io/mailhog/mailhog:v1.0.1
INFO[0005] Retrieving image docker.io/mailhog/mailhog:v1.0.1 from registry index.docker.io
INFO[0015] Retrieving image manifest docker.server/drupal/php:-dev
INFO[0015] Retrieving image docker.server/drupal/php:-dev from registry docker.server
WARN[0015] Error while trying to warm image: docker.server/drupal/php:-dev Failed to retrieve image: docker.server/drupal/php:-dev: GET https://docker.server/v2/drupal/php/manifests/-dev: NOT_FOUND: artifact drupal/php:-dev not found
INFO[0015] Retrieving image manifest docker.server/node:
INFO[0015] Retrieving image docker.server/node: from registry docker.server
WARN[0015] Error while trying to warm image: docker.server/node: Failed to retrieve image: docker.server/node:: GET https://docker.server/v2/node/manifests/latest: NOT_FOUND: artifact node:latest not found
INFO[0015] Retrieving image manifest docker.server/drupal/php:
INFO[0015] Retrieving image docker.server/drupal/php: from registry docker.server
WARN[0016] Error while trying to warm image: docker.server/drupal/php: Failed to retrieve image: docker.server/drupal/php:: GET https://docker.server/v2/drupal/php/manifests/latest: NOT_FOUND: artifact drupal/php:latest not found
INFO[0016] Retrieving image manifest docker.server/nginx:
INFO[0016] Retrieving image docker.server/nginx: from registry docker.server
WARN[0016] Error while trying to warm image: docker.server/nginx: Failed to retrieve image: docker.server/nginx:: GET https://docker.server/v2/nginx/manifests/latest: NOT_FOUND: artifact nginx:latest not found
No need to build image - it exists already!
Uploading artifacts for successful job

Expected behavior

Yesterday, and many many days before, the build was successful with this output:

Executing "step_script" stage of the job script 05:53
$ if warmer --image $CI_REGISTRY_IMAGE/$BUILD_NAME:$CI_COMMIT_SHORT_SHA; then echo "No need to build image - it exists already!"; exit 0; fi
INFO[0000] Retrieving image manifest gitlab.server:4567/php:ef1a2d7b
INFO[0000] Retrieving image gitlab.server:4567/php:ef1a2d7b from registry gitlab.server:4567
WARN[0000] Error while trying to warm image: gitlab.server:4567/php:ef1a2d7b Failed to retrieve image: gitlab.server:4567/php:ef1a2d7b: GET https://gitlab.server:4567/v2/php/manifests/ef1a2d7b: MANIFEST_UNKNOWN: manifest unknown; map[Tag:ef1a2d7b]
Failed warming cache: Failed to warm any of the given images
$ chmod -R o-w $CI_PROJECT_DIR
$ echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.jso
n
$ executor version
Kaniko version :  v1.11.0
$ executor --cache --context $CI_PROJECT_DIR --dockerfile $DOCKERFILE_LOCATION --target $BUILD_NAME --build-arg BUILD_ID=$CI_COMMIT_SHORT_SHA --build-arg CODE_ACCESS_TOKEN=$CODE_ACCESS_TOKEN --destination $CI_REGISTRY_IMAGE/$BUILD_NAME:$CI_COMMIT_SHORT_SHA --digest-file "${BUILD_NAME}.${IMAGE_DIGEST_EXTENSION}" --label "org.opencontainers.image.pipeline"=$CI_PIPELINE_ID --label "org.opencontainers.image.job"=$CI_JOB_ID --label "org.opencontainers.image.revision"=$CI_COMMIT_SHORT_SHA --label "org.opencontainers.image.created"=$CI_JOB_STARTED_AT $KANIKO_FLAGS

... SKIPPING UNRELATED OUTPUT ....

INFO[0341] Pushing image to gitlab.server:4567/php:ef1a2d7b
INFO[0352] Pushed gitlab.server:4567/php@sha256:70d658319367dd51ade148f32b8c4f84e9a9d25af3e908f2d4de80d3261fecbc

To Reproduce
Steps to reproduce the behavior:

$ docker run --rm -it --entrypoint=sh gcr.io/kaniko-project/executor:debug
/workspace # kaniko version
sh: kaniko: not found
/workspace # executor version
Kaniko version :  v1.11.0
/workspace # warmer --image alpine:latest
INFO[0000] Retrieving image manifest alpine:latest      
INFO[0000] Retrieving image alpine:latest from registry index.docker.io 
/workspace # echo $?
0

^D

$ docker pull gcr.io/kaniko-project/executor:debug
debug: Pulling from kaniko-project/executor
2c3172966e36: Pull complete 
1bb21d28b0fd: Pull complete 
775dac3fcce0: Pull complete 
326b84e032b7: Pull complete 
c4b3818f4c51: Pull complete 
1a0cfe34100c: Pull complete 
cc6981c0dadd: Pull complete 
5281fa3bb3fc: Pull complete 
75f45b591dd3: Pull complete 
d6712737e25d: Pull complete 
5b00811c335e: Pull complete 
795f1259ec12: Pull complete 
cae37feebc8c: Pull complete 
Digest: sha256:127b233d0ee0ebde6c4c678b129e8ed7b4655678b212e5c7da2d9738ab6b6e49
Status: Downloaded newer image for gcr.io/kaniko-project/executor:debug
gcr.io/kaniko-project/executor:debug
$ docker run --rm -it --entrypoint=sh gcr.io/kaniko-project/executor:debug
/workspace # executor version
Kaniko version :  v1.12.0
/workspace # warmer --image alpine:latest
Error: error validating dockerfile path: please provide a valid path to a Dockerfile within the build context with --dockerfile
Usage:
  cache warmer [flags]

Flags:
      --build-arg multi-arg type                  This flag should be used in conjunction with the dockerfile flag for scenarios where dynamic replacement of the base image is required.
  -c, --cache-dir string                          Directory of the cache. (default "/cache")
      --cache-ttl duration                        Cache timeout in hours. Defaults to two weeks. (default 336h0m0s)
      --customPlatform string                     Specify the build platform if different from the current host
  -d, --dockerfile string                         Path to the dockerfile to be cached. The kaniko warmer will parse and write out each stage's base image layers to the cache-dir. Using the same dockerfile path as what you plan to build in the kaniko executor is the expected usage. (default "Dockerfile")
  -f, --force                                     Force cache overwriting.
  -h, --help                                      help for cache
  -i, --image multi-arg type                      Image to cache. Set it repeatedly for multiple images. (default )
      --insecure-pull                             Pull from insecure registry using plain HTTP
      --insecure-registry multi-arg type          Insecure registry using plain HTTP to pull. Set it repeatedly for multiple registries.
      --log-format string                         Log format (text, color, json) (default "color")
      --log-timestamp                             Timestamp in log output
      --registry-certificate key-value-arg type   Use the provided certificate for TLS communication with the given registry. Expected format is 'my.registry.url=/path/to/the/server/certificate'.
      --registry-client-cert key-value-arg type   Use the provided client certificate for mutual TLS (mTLS) communication with the given registry. Expected format is 'my.registry.url=/path/to/client/cert,/path/to/client/key'.
      --registry-mirror multi-arg type            Registry mirror to use as pull-through cache instead of docker.io. Set it repeatedly for multiple mirrors.
      --skip-tls-verify-pull                      Pull from insecure registry ignoring TLS verify
      --skip-tls-verify-registry multi-arg type   Insecure registry ignoring TLS verify to pull. Set it repeatedly for multiple registries.
  -v, --verbosity string                          Log level (trace, debug, info, warn, error, fatal, panic) (default "info")

/workspace # 

Additional Information

• Kaniko Image - Digest: sha256:127b233d0ee0ebde6c4c678b129e8ed7b4655678b212e5c7da2d9738ab6b6e49

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[RoSk0]

Potentially coming from this new feature:

• feat(experimental): cache dockerfile images through warmer feat: cache dockerfile images through warmer #2499

[aaron-prindle]

Thanks you for flagging this @RoSk0, this is due to a bug in PR #2499 in which the Dockerfile path is always validated even when the flag is not supplied (it will always break for regular/previous --image only usage IIUC as it is in v1.12.0). I am going to push all of the :latest tags back to v1.11.0 and release a v1.12.1 hotfix addressing this issue. I will update this thread when this is done

[aaron-prindle]

I have bumped all named tags back to v1.11.0 for kaniko's released images. This includes:

executor
latest -> v1.11.0
debug -> debug-v1.11.0
slim -> slim-v1.11.0

warmer
latest -> v1.11.0

[aaron-prindle]

v1.12.1 released now with the fix for this