Kaniko hardcoded /bin/sh?

[isantospardo]

Actual behavior
This is the Dockerfile for the debug image builder:
https://github.com/GoogleContainerTools/kaniko/blob/master/deploy/Dockerfile_debug

it seems like the shell is installed under /busyboxy, so it would be /busybox/sh? However images understandably assume it's in /bin/sh?

docker run -it --entrypoint=/busybox/sh gcr.io/kaniko-project/executor:debug
/ # echo $PATH
/usr/local/bin:/kaniko:/busybox
/ # which sh
/busybox/sh

something in the build process is hardcoded to use /bin/sh?
I get the following error:

INFO[0088] RUN ldconfig /usr/local/lib/                 
INFO[0088] cmd: /bin/sh                                 
INFO[0088] args: [-c ldconfig /usr/local/lib/]        
gets error: error building image: error building stage: starting command: fork/exec /bin/sh: no such file or directory

[pllopis]

Additional comments about this issue:

This error appears to come from

kaniko/pkg/commands/run.go

Line 97 in 7bf3f87

return errors.Wrap(err, "starting command")

Further up in line 55 seems to be the culprit:

kaniko/pkg/commands/run.go

Line 55 in 7bf3f87

shell = append(shell, "/bin/sh", "-c")

It seems that /bin/sh -c is prepended to the command, as per the Dockerfile specification (https://docs.docker.com/engine/reference/builder/#run), which is fair enough.

However note that the Dockerfile that contains the RUN ldconfig /usr/local/lib runs locally under different Docker versions, but doesn't under Kaniko.

I tried to work around the issue by using the exec form, RUN ["/sbin/ldconfig", "/usr/local/lib/"] (the Dockerfile is using FROM golang, which means ldconfig is in that path) but then Kaniko complains about /sbin/ldconfig not existing.

NFO[0075] RUN ["/sbin/ldconfig", "/usr/local/lib/"]    
INFO[0075] cmd: /sbin/ldconfig                          
INFO[0075] args: [/usr/local/lib/]                      
error building image: error building stage: starting command: fork/exec /sbin/ldconfig: no such file or directory

Indeed, the busybox provided with the debug image does not provide ldconfig (I thought it would run in the context of the image, using the image's /sbin/ldconfig?):

→ docker run -it --entrypoint=/busybox/sh gcr.io/kaniko-project/executor:debug
/ # which ldconfig
/ # find / -name ldconfig
/ #

It seems like something about Kaniko's implementation is making it behave different to Docker, but haven't read enough of the code to figure out how/why.

[tejal29]

Thanks @isantospardo Looks like something changed either the base image or the way we build the image right now.

I am going to dig into this deeper and fix it before the next release

[tejal29]

@isantospardo I am not able to reproduce your issue with the following dockerfile

FROM golang
RUN ["echo", "test"]
RUN ["/sbin/ldconfig", "/usr/local/lib/"] 

Here is the log.
Can you please double-check again?
I verified the above dockerfile on latest master, "gcr.io/kaniko-project/executor:debug-v0.16.0" and "gcr.io/kaniko-project/executor:debug"

tejaldesai@@skaffold (prototype)$ docker run -it --entrypoint /busybox/sh -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/tejal-test/executor:debug
/ # /kaniko/executor -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test --tarPath=image.tar
INFO[0000] Resolved base name golang to golang          
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Resolved base name golang to golang          
INFO[0000] Retrieving image manifest golang             
INFO[0001] Retrieving image manifest golang             
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest golang             
INFO[0002] Retrieving image manifest golang             
INFO[0003] Unpacking rootfs as cmd RUN ["echo", "test"] requires it. 
INFO[0015] Taking snapshot of full filesystem...        
INFO[0015] Resolving paths                              
INFO[0019] RUN ["echo", "test"]                         
INFO[0019] cmd: echo                                    
INFO[0019] args: [test]                                 
test
INFO[0019] Taking snapshot of full filesystem...        
INFO[0019] Resolving paths                              
INFO[0022] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0022] RUN ["/sbin/ldconfig", "/usr/local/lib/"]    
INFO[0022] cmd: /sbin/ldconfig                          
INFO[0022] args: [/usr/local/lib/]                      
INFO[0022] Taking snapshot of full filesystem...        
INFO[0022] Resolving paths                              
/ # 

[tejal29]

@isantospardo Can you please verify?

[pllopis]

Here's a simplified Dockerfile that reproduces this issue. Works with Docker but doesn't with Kaniko:

FROM golang

RUN mkdir /something
WORKDIR /something
RUN rm -rf /something
RUN ["/sbin/ldconfig", "/usr/local/lib/"]

Which yields:

INFO[0003] Downloading base image golang                
 2020/03/24 10:29:14 No matching credentials were found, falling back on anonymous
 INFO[0004] Unpacking rootfs as cmd RUN mkdir /something requires it. 
 INFO[0028] Taking snapshot of full filesystem...        
 INFO[0039] RUN mkdir /something                         
 INFO[0039] cmd: /bin/sh                                 
 INFO[0039] args: [-c mkdir /something]                  
 INFO[0039] Taking snapshot of full filesystem...        
 INFO[0045] WORKDIR /something                           
 INFO[0045] cmd: workdir                                 
 INFO[0045] Changed working directory to /something      
 INFO[0045] RUN rm -rf /something                        
 INFO[0045] cmd: /bin/sh                                 
 INFO[0045] args: [-c rm -rf /something]                 
 INFO[0045] Taking snapshot of full filesystem...        
 INFO[0045] Adding whiteout for /something               
 INFO[0050] RUN ["/sbin/ldconfig", "/usr/local/lib/"]    
 INFO[0050] cmd: /sbin/ldconfig                          
 INFO[0050] args: [/usr/local/lib/]                      
 error building image: error building stage: starting command: fork/exec /sbin/ldconfig: no such file or directory
 ERROR: Job failed: exit code 1

However if you comment out that line that does RUN rm -rf /something, it works:

INFO[0002] Downloading base image golang                
 2020/03/24 10:29:41 No matching credentials were found, falling back on anonymous
 INFO[0003] Error while retrieving image from cache: getting file info: stat /cache/sha256:5265bbb65c02bb07591cbf2044eadbf900ac25f5c47b2dcd2d42eba403d2a525: no such file or directory 
 INFO[0003] Downloading base image golang                
 2020/03/24 10:29:42 No matching credentials were found, falling back on anonymous
 INFO[0004] Unpacking rootfs as cmd RUN mkdir /something requires it. 
 INFO[0028] Taking snapshot of full filesystem...        
 INFO[0036] RUN mkdir /something                         
 INFO[0036] cmd: /bin/sh                                 
 INFO[0036] args: [-c mkdir /something]                  
 INFO[0036] Taking snapshot of full filesystem...        
 INFO[0041] WORKDIR /something                           
 INFO[0041] cmd: workdir                                 
 INFO[0041] Changed working directory to /something      
 INFO[0041] RUN ["/sbin/ldconfig", "/usr/local/lib/"]    
 INFO[0041] cmd: /sbin/ldconfig                          
 INFO[0041] args: [/usr/local/lib/]                      
 INFO[0041] Taking snapshot of full filesystem...    

[pllopis]

A possible workaround that also makes it work is to change the workdir back to a directory that exists, such as:

WORKDIR /

In my Dockerfile I was using this pattern of ADD something.tgz, WORKDIR /something, RUN ./configure && make install, RUN rm -rf /something, the rm -rf being to free up unnecessary data. Then I was using this as a base image in another Dockerfile which runs the ldconfig and triggers this bug (which works in Docker). Thankfully the issue can be simplified to be reproduced with a single Dockerfile.

Thanks for looking into this!

[olivierboudet]

Hello, I have the same issue with kaniko 0.19.0
Other simple Dockerfile to reproduce :

FROM gcr.io/go-containerregistry/gcrane:debug
RUN useradd --create-home --uid 1000 jenkins

INFO[0001] Resolved base name gcr.io/go-containerregistry/gcrane:debug to gcr.io/go-containerregistry/gcrane:debug 
INFO[0001] Resolved base name gcr.io/go-containerregistry/gcrane:debug to gcr.io/go-containerregistry/gcrane:debug 
INFO[0001] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0002] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0003] Built cross stage deps: map[]                
INFO[0003] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0004] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0005] Checking for cached layer gcr.io/xxx-xxx/xxx/builder-test/cache:4b02ce20c25c9dc6580b2c63633c1210083bf935679c19e8a66de86f58066168... 
INFO[0005] No cached layer found for cmd RUN useradd --create-home --uid 1000 jenkins 
INFO[0005] Unpacking rootfs as cmd RUN useradd --create-home --uid 1000 jenkins requires it. 
INFO[0017] Taking snapshot of full filesystem...        
INFO[0017] Resolving paths                              
INFO[0017] RUN useradd --create-home --uid 1000 jenkins 
INFO[0017] cmd: /bin/sh                                 
INFO[0017] args: [-c useradd --create-home --uid 1000 jenkins] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory

[tejal29]

Thanks. Understood.

[tejal29]

I was not able to reproduce your use case @olivierboudet

/ # /kaniko/executor --context=dir://workspace --no-push
INFO[0000] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0000] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0001] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0002] Executing 0 build triggers                   
INFO[0002] Unpacking rootfs as cmd RUN useradd --create-home --uid 1000 jenkins requires it. 
INFO[0004] RUN useradd --create-home --uid 1000 jenkins 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] Resolving 28165 paths                        
INFO[0007] cmd: /bin/sh                                 
INFO[0007] args: [-c useradd --create-home --uid 1000 jenkins] 
INFO[0007] Running: [/bin/sh -c useradd --create-home --uid 1000 jenkins] 
INFO[0007] Taking snapshot of full filesystem...        
INFO[0007] Resolving 28171 paths                        
INFO[0009] Skipping push to container registry due to --no-push flag 

It could also be because, the gcr.io/go-containerregistry/gcrane:debug has an empty /bin/ which get extracted and overridden.
Can you file a bug with them?

[olivierboudet]

I was not able to reproduce your use case @olivierboudet

Ok, I tested again with kaniko 0.22.0, I reproduced it.

$ cat Dockerfile
FROM gcr.io/go-containerregistry/gcrane:debug
RUN useradd --create-home --uid 1000 jenkins

$ docker run -v $(pwd):/workspace gcr.io/kaniko-project/executor:debug-v0.22.0 --dockerfile ./Dockerfile --no-push --context dir:///workspace/
INFO[0000] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0000] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0002] Retrieving image manifest gcr.io/go-containerregistry/gcrane:debug 
INFO[0002] Executing 0 build triggers                   
INFO[0002] Unpacking rootfs as cmd RUN useradd --create-home --uid 1000 jenkins requires it. 
INFO[0014] RUN useradd --create-home --uid 1000 jenkins 
INFO[0014] Taking snapshot of full filesystem...        
INFO[0014] Resolving 2481 paths                         
INFO[0014] cmd: /bin/sh                                 
INFO[0014] args: [-c useradd --create-home --uid 1000 jenkins] 
INFO[0014] Running: [/bin/sh -c useradd --create-home --uid 1000 jenkins] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory

I will open an issue with gcrane but this dockerfile is fully building with docker deamon.

[siscia]

I got a similar problem that @olivierboudet

It can be reproduced also with cern/slc6-base.

I solved the problem by re-setting the PATH environment variable inside the dockerfile.
Like this: https://gitlab.cern.ch/cernvm/build-images/-/commit/57ce9d957309253a570aa1fa42e0c3c2bb06d197

[lishisongabthony]

最终解决方案是啥 啊

[Erick-shi]

error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory

so, just write PATH in Dockerfile? but I'm failed....