[Flake] Cloud Build push-images fails sometimes (windows binary missing)

[markmandel]

What happened:

Example:
#2789 (comment)

#10 [auth] agones-images/ci/agones-sdk:pull,push token for us-docker.pkg.dev
#10 DONE 0.0s
docker run --rm -e "GOOS=darwin" -e "GOARCH=arm64" -e "GO111MODULE=on" -w /go/src/agones.dev/agones -v /workspace/build//.config/gcloud:/root/.config/gcloud -v ~/.kube/:/root/.kube -v ~/.config/helm:/root/.config/helm -v ~/.cache/helm:/root/.cache/helm -v /workspace:/go/src/agones.dev/agones -v /workspace/build//.gomod:/go/pkg/mod -v /workspace/build//.gocache:/root/.cache/go-build agones-build:d3c559f349 go build -mod=vendor \
	-o /go/src/agones.dev/agones/cmd/sdk-server/bin/sdk-server.darwin.arm64  -ldflags "-X agones.dev/agones/pkg.Version=1.28.0-a08ed89" agones.dev/agones/cmd/sdk-server
docker build /workspace/cmd/controller/ --tag=us-docker.pkg.dev/agones-images/ci/agones-controller:1.28.0-a08ed89-amd64 
#8 DONE 0.3s
docker build /workspace/cmd/sdk-server/ --tag=us-docker.pkg.dev/agones-images/ci/agones-sdk:1.28.0-a08ed89-linux-amd64 
error checking context: 'file ('/workspace/cmd/sdk-server/bin/sdk-server.windows.amd64.exe') not found or excluded by .dockerignore'.
make: *** [Makefile:492: build-agones-sdk-image-amd64] Error 1
make: *** Waiting for unfinished jobs....

What you expected to happen:

The window binary should be there. Not sure why it isn't. Need to dig in and see what's happening.

Feels like a weird race condition.

How to reproduce it (as minimally and precisely as possible):

I expect you could reproduce this by running make -j 4 push-images, or check for it in CI.

Anything else we need to know?:

I have a feeling this is the issue:

agones/build/Makefile

Lines 601 to 604 in fc9727d

	push-agones-sdk-windows:
	# Using docker buildx with foreign OSes is a bit awkward. Since the layers but not the tag
	# was cached we have to rebuild the image (which should be fast since it's cached).
	$(MAKE) WINDOWS_DOCKER_PUSH_ARGS=--push build-agones-sdk-image-windows

Since we invoke make directly, it can't track the build-agones-sdk-binary across all the targets in make push and make it parallel as appropriate - so the binaries get built twice, and I'm willing to bet, in some circumstances, overwrite each other.

Environment:

• Agones version: dev
• Kubernetes version (use kubectl version): N/A
• Cloud provider or hardware configuration: N/A
• Install method (yaml/helm): N/A
• Troubleshooting guide log(s): See above.
• Others:

[markmandel]

Just writing this down so I don't forget to test it later, I think you could do:

push-agones-sdk-windows: WINDOWS_DOCKER_PUSH_ARGS=--push
push-agones-sdk-windows: build-agones-sdk-image-windows 
push-agones-sdk-windows:

And I think that WINDOWS_DOCKER_PUSH_ARGS will pass through? 🤔 and the dependency graph will work correctly.