Remove tarball.WithCompressedCaching flag to resolve OOM Killed error #1722
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed (or fixed any issues), please reply here with What to do if you already signed the CLAIndividual signers
Corporate signers
ℹ️ Googlers: Go here for more info. |
|
@googlebot I signed it! |
google-cla
bot
added
cla: yes
|
I just improved my PR by adding a command line flag. Now it is possible to set I really hope, that one of the maintainer can pick this up. |
tejal29
approved these changes
Oct 19, 2021
Large images cannot be build as the kaniko container will be killed due to an OOM error. Removing the tarball compression drastically reduces the memory required to push large image layers. Fixes GoogleContainerTools#1680 This change may increase the build time for smaller images. Therefore a command line option to trigger the compression or a more intelligent behaviour may be useful.
tejal29
force-pushed
the
1680-oom-killed
branch
from
b80c225
to
78fe2f5
Compare
|
Rebased over latest master! Will merge and pick up for release. |
stefannica
added a commit
to stefannica/extensions
that referenced
this pull request
Nov 15, 2021
If the k8s node where the MLFlow builder step is running doesn't
have a lot of memory, the builder step will fail if it has to build
larger images. For example, building the trainer image for the keras
CIFAR10 codeset example resulted in an OOM failure on a node where
only 8GB of memory were available.
This is a known kaniko issue [1] and there's a fix available [2] with
more recent (>=1.7.0) kaniko versions: disabling the compressed
caching via the `--compressed-caching` command line argument.
This commit models a workflow input parameter mapped to this
new command line argument. To avoid OOM errors with bigger
images, the user may set it in the workflow like so:
```
- name: builder
image: ghcr.io/stefannica/mlflow-builder:latest
inputs:
- name: mlflow-codeset
codeset:
name: '{{ inputs.mlflow-codeset }}'
path: /project
- name: compressed_caching
# Disable compressed caching to avoid running into OOM errors on cluster nodes with lower memory
value: false
```
[1] GoogleContainerTools/kaniko#909
[2] GoogleContainerTools/kaniko#1722
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Large images cannot be build as the kaniko container will be killed due to an OOM error. Removing the tarball compression drastically reduces the memory required to push large image layers. Fixes #1680
Removing the tarball compression may increase the build time for smaller images. Therefore a command line option to disable the compression was chosen for the implementation.
Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:
I may need some support here, generating a good integration test.
Currently the integration tests are failing due to: Fix executor Dockerfile, which wasn't building #1741
See the contribution guide for more details.
Reviewer Notes
Release Notes