auto-bless tests/formatting on rust-lang/rust

[Mark-Simulacrum]

This is intended to be somewhat compatible with other repositories wishing to do the same, but for now it is reasonable to hardcode assumptions that can be made for rust-lang/rust.

I believe these are reasonable steps to making this work:

Adjust compiletest to have a --bless-into $DIR flag which acts like --bless but writes new files at a different root (e.g., if editing src/test/ui/issue-3433.stderr, we'd instead save to $DIR/src/test/ui/issue-3433.stderr). It should not change the status reported by tests, merely write new files (i.e., it is safe to always pass this on CI).

We will want to do the same for rustfmt blessing; it is not entirely clear what the best way to do this is though. We wouldn't want to write all files to the new directory, just those that have changed when rustfmt is applied.

Once both of these are done, we will want to fetch this directory from CI somehow. I suspect we want this to be in the PR CI job so that everyone can use it (vs. waiting for bors to fail, which seems suboptimal). Eventually we'll want to add to the bors builders as well, so that all bless-able failures can be resolved via this bot.

This is somewhat hard -- we can't readily grant PR CI keys to write into an S3 bucket or anything like that. I suspect that a good way to do this might be GitHub Action's artifacts. AFAICT, we get a unique location guaranteed by GitHub via these artifacts so they should be safe to use.

Once all of that is done, we will already have a manual way to bless without building Rust: download the zip file from GHA, and "merge" in the new files locally, and push up a commit. This step can be automated via triagebot as well without too much trouble (though identifying the workflow ID is IIRC annoying).

[Mark-Simulacrum]

@lcnr you expressed interested in working on this, so pinging you.

[jyn514]

"merge" in the new files locally

We shouldn't need to merge the files, just overwrite them with the new ones, right?

[Mark-Simulacrum]

Right, yes, I just mean that the downloaded zipfile contains a subset of the files rather than all of them.

[lcnr]

Adjust compiletest to have a --bless-into $DIR flag which acts like --bless but writes new files at a different root (e.g., if editing src/test/ui/issue-3433.stderr, we'd instead save to $DIR/src/test/ui/issue-3433.stderr). It should not change the status reported by tests, merely write new files (i.e., it is safe to always pass this on CI).

We already save these files to /build/x86_64-unknown-linux-gnu/test/ui/, so it might make sense to instead just read them from there 🤔 Is there something that makes using these files more difficult than writing them in a seperate dir?

[Mark-Simulacrum]

Well, no, not particularly - that should be fine. It might be a bit more annoying to construct that path on different builders, but probably not too hard - we really only need 1 or 2 builders for pretty complete coverage anyway.

[lcnr]

Looks like this is slightly too complex for me to implement with my current time budget, so if someone else in interested in picking this up, please do! 😬