Fix leaking of tmpfs mount in CDI mode #1168

ArangoGutierrez · 2025-06-30T09:26:35Z

This patch fixes the handling of temporary files and directories in the NVIDIA container runtime hook, ensuring we don't leak tmpfs mounts when handling the params file in the container when in CDI mode.

Refactoring and File Handling Improvements:

cmd/nvidia-cdi-hook/disable-device-node-modification/params_linux.go: Refactored createParamsFileInContainer to use a predefined scratch directory (/var/run/nvidia-ctk-hook) instead of temporary directories. Introduced the createFileInRoot function for secure file creation using securejoin and Mknodat.

Expanded Test Coverage:

tests/e2e/nvidia-container-toolkit_test.go: Added new tests to validate the behavior of disabling device node creation with both nvidia-container-runtime-hook in legacy mode and nvidia-container-runtime. These tests ensure that no device nodes are created by comparing mount states before and after running containers.

coveralls · 2025-06-30T09:28:52Z

Pull Request Test Coverage Report for Build 16003545229

Details

0 of 52 (0.0%) changed or added relevant lines in 1 file are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.09%) to 33.117%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
cmd/nvidia-cdi-hook/disable-device-node-modification/params_linux.go	0	52	0.0%

Files with Coverage Reduction	New Missed Lines	%
cmd/nvidia-cdi-hook/disable-device-node-modification/params_linux.go	2	0.0%

Totals
Change from base Build 16002648947:	-0.09%
Covered Lines:	4381
Relevant Lines:	13229

💛 - Coveralls

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>

This change ensures that the tmpfs mount created for the modified NVIDIA params file does not leak to the host. Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar · 2025-06-30T13:11:27Z

cmd/nvidia-cdi-hook/disable-device-node-modification/params_linux.go

@@ -61,3 +79,31 @@ func createParamsFileInContainer(containerRootDirPath string, contents []byte) e
 func createTmpFs(target string, size int) error {
 	return unix.Mount("tmpfs", target, "tmpfs", 0, fmt.Sprintf("size=%d", size))
 }
+
+func createFileInRoot(containerRootDirPath string, destinationPath string, mode os.FileMode) (string, error) {


TODO: This function also exists in internal/ldconfig we should move this to a separate pacakge.

elezar · 2025-06-30T13:12:36Z

tests/e2e/nvidia-container-toolkit_test.go

+			_, _, err = runner.Run("mkdir -p /tmp/empty")
+			Expect(err).ToNot(HaveOccurred())
+
+			_, _, err = runner.Run("mount | sort > /tmp/mounts.before")


What about capturing the output of mount | sort as variables and then comparing these using the Gomega matchers?

elezar · 2025-06-30T13:13:11Z

tests/e2e/nvidia-container-toolkit_test.go

@@ -234,12 +240,28 @@ var _ = Describe("docker", Ordered, ContinueOnFailure, func() {
 			Expect(err).ToNot(HaveOccurred())
 			Expect(output).To(Equal("ModifyDeviceFiles: 0\n"))
 		})
+
+		//sudo docker run --runtime=nvidia --rm -ti -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=all --mount type=bind,source=/tmp/empty,target=/empty,bind-propagation=shared  ubuntu true


What is the purpose of this comment?

oh, sorry it was me during devel, to not forget that bit

elezar · 2025-06-30T13:14:29Z

tests/e2e/nvidia-container-toolkit_test.go

@@ -234,12 +240,28 @@ var _ = Describe("docker", Ordered, ContinueOnFailure, func() {
 			Expect(err).ToNot(HaveOccurred())
 			Expect(output).To(Equal("ModifyDeviceFiles: 0\n"))
 		})
+
+		//sudo docker run --runtime=nvidia --rm -ti -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=all --mount type=bind,source=/tmp/empty,target=/empty,bind-propagation=shared  ubuntu true
+		It("should work with nvidia-container-runtime", func(ctx context.Context) {


Let's also add a case for --runtime=runc for the "legacy" code path. As a general question, could we add a tag to tests to indicate that it's targeting the legacy code path?

elezar · 2025-06-30T13:14:50Z

tests/e2e/nvidia-container-toolkit_test.go

 	})

 	When("A container is run using CDI", Ordered, func() {
 		BeforeAll(func(ctx context.Context) {
 			_, _, err := runner.Run("docker pull ubuntu")
 			Expect(err).ToNot(HaveOccurred())
+


Nit: Let's remove this from the diff.

Copilot

Pull Request Overview

This PR refactors how the NVIDIA container runtime hook creates and mounts its params file—switching to procfd-based APIs and secure file creation to prevent mount leaks—and adds end-to-end tests to ensure no host mounts remain after running a container.

Switch createParamsFileInContainer to use utils.WithProcfd and a createFileInRoot helper for safer tmpfs and bind mounts.
Introduce a secure, mknodat-based file creation function (createFileInRoot).
Add E2E tests that record host mounts before/after running containers in both legacy and nvidia runtimes to catch mount leaks.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
tests/e2e/nvidia-container-toolkit_test.go	New “Disabling device node creation” suite: captures host mounts, runs containers, and asserts no new mounts
cmd/nvidia-cdi-hook/disable-device-node-modification/params_linux.go	Refactored `createParamsFileInContainer` to use procfd mounts and secure file creation, replacing temp dir logic

Comments suppressed due to low confidence (1)

cmd/nvidia-cdi-hook/disable-device-node-modification/params_linux.go:44

[nitpick] The error message could include the target path (e.g., hookScratchDirPath) to make debugging mount failures clearer.

		return fmt.Errorf("failed to create tmpfs mount for params file: %w", err)

Copilot · 2025-07-01T07:46:56Z

cmd/nvidia-cdi-hook/disable-device-node-modification/params_linux.go

-	err = utils.WithProcfd(containerRootDirPath, nvidiaDriverParamsPath, func(nvidiaDriverParamsFdPath string) error {
-		return unix.Mount(modifiedParamsFile.Name(), nvidiaDriverParamsFdPath, "", unix.MS_BIND|unix.MS_RDONLY|unix.MS_NODEV|unix.MS_PRIVATE|unix.MS_NOSYMFOLLOW, "")
+	err = utils.WithProcfd(containerRootDirPath, modifiedParamsFilePath, func(modifiedParamsFileFdPath string) error {
+		modifiedParamsFile, err := os.OpenFile(modifiedParamsFileFdPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0444)


The opened file descriptor for modifiedParamsFile is never closed, which can lead to resource leaks. Add a defer modifiedParamsFile.Close() immediately after opening.

Copilot · 2025-07-01T07:46:57Z

cmd/nvidia-cdi-hook/disable-device-node-modification/params_linux.go

-	tmpRoot, err := os.MkdirTemp("", "nvct-empty-dir*")
-	if err != nil {
-		return fmt.Errorf("failed to create temp root: %w", err)
+	hookScratchDirPath := "/var/run/nvidia-ctk-hook"


[nitpick] This magic path (/var/run/nvidia-ctk-hook) should be extracted to a package-level constant to improve clarity and avoid duplication.

Suggested change

hookScratchDirPath := "/var/run/nvidia-ctk-hook"

Copilot · 2025-07-01T07:46:57Z

tests/e2e/nvidia-container-toolkit_test.go

+			_, _, err = runner.Run("mkdir -p /tmp/empty")
+			Expect(err).ToNot(HaveOccurred())
+
+			mountsBefore, _, err = runner.Run("mount | sort")
+			Expect(err).ToNot(HaveOccurred())
+		})
+
+		AfterAll(func() {
+			_, _, err := runner.Run("rm -rf /tmp/empty")
 			Expect(err).ToNot(HaveOccurred())


[nitpick] Using a fixed path (/tmp/empty) may cause collisions or flaky tests. Consider creating a unique temporary directory (e.g., via Ginkgo's TempDir() or mktemp) and cleaning it up afterward.

Suggested change

_, _, err = runner.Run("mkdir -p /tmp/empty")

Expect(err).ToNot(HaveOccurred())

mountsBefore, _, err = runner.Run("mount | sort")

Expect(err).ToNot(HaveOccurred())

})

AfterAll(func() {

_, _, err := runner.Run("rm -rf /tmp/empty")

Expect(err).ToNot(HaveOccurred())

tempDir := TempDir("", "empty")

mountsBefore, _, err = runner.Run("mount | sort")

Expect(err).ToNot(HaveOccurred())

})

AfterAll(func() {

// No explicit cleanup needed as TempDir() handles it automatically

… in container Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>

ArangoGutierrez requested review from elezar and Copilot June 30, 2025 09:26

ArangoGutierrez self-assigned this Jun 30, 2025

This comment was marked as outdated.

Sign in to view

ArangoGutierrez added the bug Issue/PR to expose/discuss/fix a bug label Jun 30, 2025

ArangoGutierrez force-pushed the b/5363680 branch 2 times, most recently from 1b9c576 to 31828a6 Compare June 30, 2025 10:59

Add E2E test for Disabling device node creation checks

eba638f

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>

ArangoGutierrez force-pushed the b/5363680 branch 2 times, most recently from 38c25d5 to 1c79350 Compare June 30, 2025 12:00

elezar force-pushed the b/5363680 branch from 1c79350 to 05923b3 Compare June 30, 2025 12:49

elezar reviewed Jun 30, 2025

View reviewed changes

ArangoGutierrez changed the title ~~Fix Disabling device node creation hook by passing MS_PRIVATE flag during mount creation~~ Fix handling of temporary files and directories in the NVIDIA container runtime hook Jun 30, 2025

ArangoGutierrez changed the title ~~Fix handling of temporary files and directories in the NVIDIA container runtime hook~~ Fix createParamsFileInContainer func to prevent mount leaks when calling the NVIDIA container runtime hook Jun 30, 2025

ArangoGutierrez force-pushed the b/5363680 branch 2 times, most recently from 50ac666 to 91f1a73 Compare June 30, 2025 15:48

ArangoGutierrez requested review from elezar and Copilot July 1, 2025 07:45

Copilot AI reviewed Jul 1, 2025

View reviewed changes

ArangoGutierrez changed the title ~~Fix createParamsFileInContainer func to prevent mount leaks when calling the NVIDIA container runtime hook~~ Fix leaking of tmpfs mount in CDI mode Jul 1, 2025

[no-relnote] Pass flag MS_PRIVATE when creating mount for params file…

70cb383

… in container Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>

ArangoGutierrez force-pushed the b/5363680 branch from 91f1a73 to 70cb383 Compare July 1, 2025 15:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix leaking of tmpfs mount in CDI mode #1168

Fix leaking of tmpfs mount in CDI mode #1168

Uh oh!

ArangoGutierrez commented Jun 30, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

coveralls commented Jun 30, 2025 •

edited

Loading

Uh oh!

elezar Jun 30, 2025

Uh oh!

elezar Jun 30, 2025

Uh oh!

elezar Jun 30, 2025

Uh oh!

ArangoGutierrez Jun 30, 2025

Uh oh!

elezar Jun 30, 2025

Uh oh!

elezar Jun 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jul 1, 2025

Uh oh!

Copilot AI Jul 1, 2025

Uh oh!

Copilot AI Jul 1, 2025

Uh oh!

Uh oh!

Fix leaking of tmpfs mount in CDI mode #1168

Are you sure you want to change the base?

Fix leaking of tmpfs mount in CDI mode #1168

Uh oh!

Conversation

ArangoGutierrez commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Refactoring and File Handling Improvements:

Expanded Test Coverage:

Uh oh!

This comment was marked as outdated.

Uh oh!

coveralls commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 16003545229

Details

💛 - Coveralls

Uh oh!

elezar Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

elezar Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

elezar Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

ArangoGutierrez Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

elezar Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

elezar Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ArangoGutierrez commented Jun 30, 2025 •

edited

Loading

coveralls commented Jun 30, 2025 •

edited

Loading