node-daily-v14.x-staging is failing

[richardlau]

https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-v14.x-staging/ has been failing since 6 December. Cursory glance appears to be:

On macOS

macOS 10.14 builds cannot find a C compiler:
e.g. https://ci.nodejs.org/job/node-test-commit-osx/49308/nodes=osx1014/console

11:00:31 python ./configure --verbose 
11:00:32 Node.js configure: Found Python 2.7.10...
11:00:32 Detected C++ compiler (CXX=/usr/local/bin/ccache c++) version: 
11:00:32 WARNING: failed to autodetect C++ compiler version (CXX=/usr/local/bin/ccache c++)
11:00:32 Detected C compiler (CC=/usr/local/bin/ccache cc) version: 
11:00:32 WARNING: failed to autodetect C compiler version (CC=/usr/local/bin/ccache cc)
11:00:32 ERROR: No acceptable C compiler found!
11:00:32 
11:00:32        Please make sure you have a C compiler installed on your system and/or
11:00:32        consider adjusting the CC environment variable if you installed
11:00:32        it in a non-standard prefix.
11:00:32 make: *** [build-ci] Error 1

while macOS 10.15 builds fail to compile addons:
e.g. https://ci.nodejs.org/job/node-test-commit-osx/49308/nodes=osx1015/console

12:12:11 No receipt for 'com.apple.pkg.CLTools_Executables' found at '/'.
12:12:11 
12:12:11 No receipt for 'com.apple.pkg.DeveloperToolsCLILeo' found at '/'.
12:12:11 
12:12:11 No receipt for 'com.apple.pkg.DeveloperToolsCLI' found at '/'.
12:12:11 
12:12:11 gyp: No Xcode or CLT version detected!
12:12:11 gyp ERR! configure error 
12:12:11 gyp ERR! stack Error: `gyp` failed with exit code: 1

both of these are likely to be because the rebuilt VM's will need the steps documented in https://github.com/nodejs/build/blob/main/ansible/MANUAL_STEPS.md#install-command-line-tools-for-xcode applied to install a detectable by the version of gyp in Node.js 14 and old versions of node-gyp in npm 6.

On smartos there's one or more stray Node.js process being left behind:

11:29:41 ps awwx | grep Release/node | grep -v grep | cat
11:29:42  574381 ?        S  0:01 /home/iojs/build/workspace/node-test-commit-smartos/nodes/smartos20-64/out/Release/node -e setInterval(()=>{}, 99)
11:29:42  574413 ?        S  0:01 /home/iojs/build/workspace/node-test-commit-smartos/nodes/smartos20-64/out/Release/node -e setInterval(()=>{}, 99)
11:29:42 make[1]: *** [Makefile:515: test-ci] Error 1

Needs further investigation. The CI jobs are configured to fail if leftover Node.js processes are detected. As far as I can tell the tests themselves have passed. This may have started after the recent migration of the smartos machines (cc @nodejs/platform-smartos @bahamat @sxa ) -- I had clean CI's for Node.js 14.21.2 on December 8: https://ci.nodejs.org/job/node-test-commit-smartos/46911/nodes=smartos20-64/

[UlisesGascon]

@richardlau I think that we can retry the builds for macos, I just finished to fully re-ansible the Orka machines. So by now we should have the environment ready

[richardlau]

@UlisesGascon You can retry them (or start new ones) -- just set the GIT_REMOTE_REF parameter of node-test-commit-osx to refs/heads/v14.x-staging. I suspect it will still fail -- the Ansible scripts can't run those manual steps as it requires an Apple Developer account to download xcode onto the machines.

[targos]

it requires an Apple Developer account to download xcode onto the machines.

Not sure about that. On test-nearform-macos10.15-x64-3, I was able to install the command line tools for XCode without any Apple account. I used the system updates GUI though (the CLI script failed with an error that I don't exactly remember, but it was basically saying that the component we tried to install didn't exist)

[richardlau]

@targos the issue is that it needs to be installed in a certain way otherwise the older version of gyp cannot detect it.

[richardlau]

The macOS 10.14 issue was that the ccache symlink (/usr/local/bin/ccache) doesn't exist on test-orka-macos10.14-x64-2 (but does on test-orka-macos10.14-x64-1).

[richardlau]

For macOS 10.15, pkgutil --pkg-info=com.apple.pkg.CLTools_Executables works on:

• test-nearform-macos10.15-x64-1
• test-nearform-macos10.15-x64-3

and fails everywhere else.

Updating:

• test-nearform-macos10.15-x64-2
• test-orka-macos10.15-x64-1
• test-orka-macos10.15-x64-2

[richardlau]

macOS Node.js 14 CI is now fixed. Just the smartos issue left.

[richardlau]

Opened #3154 for smartos and going to close this one as the macos builds were fixed.