git-rebase job failing with fatal: bad object errors

[richardlau]

node-test-commit-arm-fanned and node-test-commit-windows-fanned are currently broken due to errors running git-rebase.

Refs: nodejs/node#49155 (comment)

The git-rebase job failed 🤔

13:05:02 + git push binary_tmp@147.75.72.255:binary_tmp.git +jenkins-node-test-commit-windows-fanned-e71525d18c4b7f2241bdf6654d7993930d51bcfd
13:05:03 remote: fatal: bad object refs/heads/jenkins-node-test-commit-arm-fanned-44627007be7eb3934e39bdd3df7e8f2556a2cd44-binary-pi1p/cc-armv7        
13:05:03 fatal: bad object refs/heads/jenkins-node-test-commit-arm-fanned-44627007be7eb3934e39bdd3df7e8f2556a2cd44-binary-pi1p/cc-armv7
13:05:03 To 147.75.72.255:binary_tmp.git
13:05:03  ! [remote rejected]         jenkins-node-test-commit-windows-fanned-e71525d18c4b7f2241bdf6654d7993930d51bcfd -> jenkins-node-test-commit-windows-fanned-e71525d18c4b7f2241bdf6654d7993930d51bcfd (missing necessary objects)
13:05:03 error: failed to push some refs to 'binary_tmp@147.75.72.255:binary_tmp.git'

The bad object (refs/heads/jenkins-node-test-commit-arm-fanned-44627007be7eb3934e39bdd3df7e8f2556a2cd44-binary-pi1p/cc-armv7) is not the one for the rebase job running (jenkins-node-test-commit-windows-fanned-e71525d18c4b7f2241bdf6654d7993930d51bcfd). The bad object appears to be related to nodejs/node#49150 (at least nodejs/node@4462700 belongs to that PR).

We had something similar a few years ago in #2153 -- we'll probably need to delete the bad object ref from the internal temp binary git repository we use (see #2153 (comment) for an example -- you may need to use sudo to become the binary_tmp user). 147.75.72.255 is jenkins-workspace-7 -- the git repository should be in /home/iojs/build/binary_tmp.git: https://github.com/nodejs/build/blob/ae3165ad2fa915564a23e1e50be9ca934b2dff0d/ansible/inventory.yml#L148C54-L148C61

Unfortunately the timing isn't great and I won't be able to log into the machine until Tuesday. I think anyone with test level access should be able to log into the machine. cc @nodejs/build

[MoLow]

I am trying to handle this

[MoLow]

I was not able to initially log in to binary_tmp, so I had to change its default shell from /usr/bin/git-shell to /bin/bash
Ran clean_binary_tmp.sh and received fatal: bad object refs/heads/jenkins-node-test-commit-arm-fanned-44627007be7eb3934e39bdd3df7e8f2556a2cd44-binary-pi1p/cc-armv7

then ran git update-ref -d refs/heads/jenkins-node-test-commit-arm-fanned-44627007be7eb3934e39bdd3df7e8f2556a2cd44-binary-pi1p/cc-armv7 and clean_binary_tmp.sh seems to be working now. will re-run the CI to confirm the issue was resolved

[MoLow]

issue seems to be resolved, for both windows and arm:
https://ci.nodejs.org/job/git-rebase/100954/
https://ci.nodejs.org/job/git-rebase/100955/

[richardlau]

Thanks @MoLow for fixing this while I was out 👍.