GitHub check for "failures all known"

[jkoritzinsky]

A few times in CI Council we've discussed how we enforce that developers reference the tracking issues for failing tests.

One possible way to enforce this would be to hook into the runfo triage issues and produce a GitHub Check on PRs to track the failing tests. The check would be green there were no failures or if every job (for Timeline queries) or test (for Test, Helix Log, etc. queries) that failed matched an open triage issue. Otherwise, the check would be red. This would enable better tracking for ensuring that PRs correctly reference why there are failing tests before merging on red.

[jaredpar]

I've had similar thoughts. I do think whatever system we decide on for tracking test failures is going to have a behavior similar to this. Probably the simplest approach to start with is a bot that just opens a comment box detailing the failures which are known / not known.