Add GH workflow to build and release on PyPI (WIP)

[lukpueh]

Fixes #1550

This PR adds a workflow with two jobs to build and release on PyPI. The release job waits for the build job and uses a custom release environment, which can be configured to require review.

To share the build artifacts between the jobs and to make them available for intermediate review, we use actions/upload-artifact (in build) and actions/download-artifact (in release) (also see GitHub docs).

To upload the build artifacts to PyPI, we use pypa/gh-action-pypi-publish as recommend by PyPA.

A demo is available in my fork.

Caveat
The URL to grab the build artifacts created in a workflow run requires knowledge of the corresponding action ID and artifact ID, plus a login token (w/o special permissions). This makes it a bit cumbersome to fetch the artifacts with a script in order to compare them to a local build. It might be easier to instruct reviewer to download the artifacts manually using the GitHub UI, and tailor the local verification script accordingly.

TODO

• Change repository_url to release on PyPI instead of Test PyPI and update the token
• Require successful CI run for a pushed tag to trigger the workflow, instead of just a push (see workflow_run docs)
• Integrate workflow with local verification (see caveat above)
• Add "Release on GitHub" job to workflow
  Note, this could be akin to and in parallel with the "release-on-pypi" job, waiting for the same review approval, and uploading the same build artifacts as release assets. Alternatively, we could create the release as part of the build job and upload the build artifacts as release assets right away, bypassing the download/upload caveat from above. It would be nice if we could set the release as draft at first, but that has the same caveat, i.e. it requires a token and special knowledge to download the assets.
• Update documentation
  • RELEASE.md
  • INSTALLATION.html#verify-release-signatures

[coveralls]

Pull Request Test Coverage Report for Build 2076681489

• 1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.001%) to 98.311%

---

Totals
Change from base Build 2057963481:	-0.001%
Covered Lines:	1178
Relevant Lines:	1194

---

💛 - Coveralls

[lukpueh]

Re "Release on GitHub":
Unfortunately, the GitHub-native action/create-release is unmaintained. But there is a third-party softprops/action-gh-release, which seems popular. If we use that we'd have to make an exception in our "only allow actions created by GitHub"-policy.

[jku]

The URL to grab the build artifacts created in a workflow run requires knowledge of the corresponding action ID and artifact ID, plus a login token

😮 wow. They really are making artifacts hidden from unauthenticated users, how disappointing of GitHub. This limits the number of people able to download the artifacts to a mere 70 million accounts: a security improvement indeed 🙄

This was so close to Just Working but then... not quite.

So, assuming we do want to integrate the GitHub release into the process and also want to have the local verification as automatic as possible, a potential solution for both is:

• add code to do the GH release (either using a 3rd party action or just adding the code, it doesn't look horribly bad for our simple case, 1 call to create release, 1 call for each asset)
• do the GH release during the "build" job (as in before the maintainer deploy permission), but name the release like "1.1.0-rc" or something
• now verify_release would be able to check that the release (candidate) matches a local build
• after maintainer approval, the "deploy" job would take the release artifacts and push them to pypi. GH release can be renamed to "1.1.0"

It's definitely more work than I expected but maybe doable? We would not need to handle edge cases since we could just say e.g. that maintainers need to delete failed -rc releases before trying to re-release

[jku]

I think using the pypa/gh-action-pypi-publish action seems fine... but also it literally just calls twine check and twine upload and that's it. We could do the same manually (and this would allow running twine check in the build phase instead of failing in deploy)

[jku]

release environment, which can be configured to require review.

To clarify, this means configured in GitHub project/environment settings, not in the workflow?

Are we defining that all maintainers can deploy? I think this might be fine: Current pypi.org project maintainers is a subset of all maintainers but that makes sense as you can upload anything with pypi.org access. With deploy permission you can only release from git sources...

[lukpueh]

😮 wow. They really are making artifacts hidden from unauthenticated users, how disappointing of GitHub. This limits the number of people able to download the artifacts to a mere 70 million accounts: a security improvement indeed 🙄

Yep, others too have expressed their doubt about the security benefit of this restriction.

This was so close to Just Working but then... not quite.

Yep, I was really excited when I discovered the download/upload feature... and then quickly somewhat disappointed.

So, assuming we do want to integrate the GitHub release into the process and also want to have the local verification as automatic as possible, a potential solution for both is:
[...]

Thanks for thinking this through with me! That's one of the approaches I had in mind. I'll explore in that direction then.

[lukpueh]

I think using the pypa/gh-action-pypi-publish action seems fine... but also it literally just calls twine check and twine upload and that's it. We could do the same manually (and this would allow running twine check in the build phase instead of failing in deploy)

Right, I forgot to mention that this is a 3rd-party action too. I was less worried since it seems endorsed by the pypa. But yeah, if it just calls twine, we might as call it ourselves.

I'll prepare a POC using pypa/gh-action-pypi-publish and softprops/action-gh-release, and in a next iteration will try to replace them with custom code.

[lukpueh]

release environment, which can be configured to require review.

To clarify, this means configured in GitHub project/environment settings, not in the workflow?

Yes! Environments and protection rules like "requires review" are configured in the GitHub repo settings (see GH docs).

Are we defining that all maintainers can deploy? I think this might be fine: Current pypi.org project maintainers is a subset of all maintainers but that makes sense as you can upload anything with pypi.org access. With deploy permission you can only release from git sources...

I haven't thought this far. I probably would have started with requiring a review from one of the maintainers who currently also has pypi.org access. I am not sure if the git sources constraint is a good argument for broadening the subset, because git sources can be anything.

[jku]

One more thing (that is probably future work as well): is signing the release still relevant?

I'm not sure if anyone uses our signatures for anything... but we could:

• have a signing key in an environment secret and sign with gpg or
• do what the cool kids are doing: https://blog.chainguard.dev/zero-friction-keyless-signing-with-github-actions/

[lukpueh]

One more thing (that is probably future work as well): is signing the release still relevant?

I'm not sure if anyone uses our signatures for anything... but we could:

• have a signing key in an environment secret and sign with gpg or
• do what the cool kids are doing: https://blog.chainguard.dev/zero-friction-keyless-signing-with-github-actions/

I'm in favor of the latter, GPG isn't relevant for PyPI anymore, and IIUC this is also the right direction towards generating in-toto attestations for our supply chain (I really need to catch up on the latest in-toto/sigstore developments).

[jku]

Even more future improvements (there's a lot of details in this one PR/discussion, no need to fix everything in one go):

For standard auditability reasons, the build log should include the hashes (and/or signatures ) of the artifacts.

[lukpueh]

Damn, I think can't re-open the PR after having deleted the remote branch in my fork, even though I have re-pushed it. Will need to check. Otherwise, I'll create a new one migrating the key points in the discussion here.