Install only CUDA runtime into images instead of entire CUDA SDK

[jiridanek]

Some (all) our images only need CUDA Runtime and do not need the rest of the CUDA SDK. Therefore, images could be made smaller if we only install the runtime. The Python packages (Tensorflow, Pythorch, ...) should only need the runtime to function.

I think the suggestion is sensible if image sizes ever become a concern. Installing everything (dnf install -y cuda-toolkit-11-8) only takes up space. I think that the tradeoff (more disk/network usage in exchange for convenience) is good for now.

This was suggested by @ben-alkov (as a general idea regarding CUDA in images, without specific investigation of the OpenDataHub situation).

Feel free to close this if no near-term action is intended to be taken on this suggestion.

[jiridanek]

@caponetto put together images size spreadsheet in https://redhat-internal.slack.com/archives/C05TTTYG599/p1721238169276599

• Maybe came the time to change the ImagePullPolicy on the notebook-controller https://issues.redhat.com/browse/RHOAIENG-10058
• Or remove the SDK for cuda and ROCm from images, leaving only runtime
• Or build images based on ubi minimal and not python3.9. Installing python 3.9 on ubi minimal ends up smaller than using the python base RHOAIENG-10350 feat(Dockerfiles): switch from s2i python images to plain ubi/c9s bases #641
• Or update minimal hardware requirements for rhoai and put there more disk space and fast internet connection as mandatory requirements.

we discussed nvidia-produced runtime images in https://redhat-internal.slack.com/archives/C06NKAPNJRX/p1721138363336889?thread_ts=1720634314.342279&cid=C06NKAPNJRX

some discussion about a test plan https://redhat-internal.slack.com/archives/D04T54X6PNV/p1711025982438499

old thread from Gabriel https://redhat-internal.slack.com/archives/C05TTTYG599/p1728482962411669

suggestion to remove some specific packages, https://redhat-internal.slack.com/archives/C05TTTYG599/p1741018570542139

iiuc, and iirc, some of that was implemented in

• include vllm package with llmcompressor #1038