Skip to content

Commit

Permalink
Nightly Release of Torch-MLIR python wheels for linux_x86_64 builds (#1)
Browse files Browse the repository at this point in the history
Reuses [build_linux_packages.sh](https://github.com/llvm/torch-mlir/blob/main/build_tools/python_deploy/build_linux_packages.sh) script to build the project and generate python wheels. I however chose to not port the original github actions workflows for snapshot releases, which seemed to do a lot more than what I initially wanted. Here's what's roughly different in this trimmed down version:
- html patching for hosting pip wheels is no longer used - instead went with the [expand_assets trick](https://fzakaria.com/2024/01/15/abusing-github-as-a-pypi-server.html), similar to [openxla/stablehlo](https://github.com/openxla/stablehlo/blob/main/.github/workflows/publishWheelRelease.yml)
- only release linux x86_64 prebuilt wheels (not macos, windows, aarch64 builds) - this may be updated later by interested parties

I used a standalone fork which was useful for the initial bringup - it took a good 30-ish iterations on `main` branch to nail the github workflow right, and some playing with auth tokens. 

Here's how the release page looks like:
- https://github.com/sjain-stanford/torch-mlir-release/releases
- https://github.com/sjain-stanford/torch-mlir-release/releases/expanded_assets/dev-wheels
(it follows the same naming convention as existing wheels, but updates the same `dev-wheels` release in place to avoid changing the expand_assets link required for pip-style hosting)

Here's the workflow: 
- https://github.com/sjain-stanford/torch-mlir-release/actions/runs/7825872338

@stellaraccident and I took a call to avoid using Personal Access Tokens (PAT) here for release permissions, and instead elevating permissions on [GITHUB_TOKEN](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#modifying-the-permissions-for-the-github_token) for `contents` scope (required for an action to make a release). We think this is reasonable considering this repo is locked down in terms of number of contributors with write access, and that this is scope specific and not a write-all.
  • Loading branch information
sjain-stanford committed Feb 8, 2024
1 parent fd84f68 commit 3c6d32e
Show file tree
Hide file tree
Showing 2 changed files with 92 additions and 0 deletions.
19 changes: 19 additions & 0 deletions .github/actions/setup-build/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
name: "Setup build environment"
description: "Setup python, ninja-build etc."

# Make top-level permissions on GITHUB_TOKEN more restrictive
permissions:
contents: read

runs:
using: "composite"

steps:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'

- name: Install prerequisites (Linux)
run: sudo apt-get install --yes ninja-build
shell: bash
73 changes: 73 additions & 0 deletions .github/workflows/buildReleaseAndPublish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
name: Build Release and Publish

on:
workflow_dispatch:
schedule:
# Runs at 11:00 AM UTC, which is 3:00 AM PST (UTC-8)
- cron: '0 11 * * *'

# Ensure that only a single job or workflow using the same
# concurrency group will run at a time. This would cancel
# any in-progress jobs in the same github workflow and github
# ref (e.g. refs/heads/main or refs/pull/<pr_number>/merge).
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
build_linux:
name: Linux x86_64 Build
runs-on: ubuntu-latest
# Elevates permissions for `GITHUB_TOKEN`'s content scope (to allow this action to make a release)
permissions:
contents: write
# Don't run this in everyone's forks.
if: github.repository == 'llvm/torch-mlir-release'
strategy:
matrix:
package: [torch-mlir]
py_version: [cp310-cp310, cp311-cp311]

steps:
- name: Checkout torch-mlir
uses: actions/checkout@v4
with:
repository: llvm/torch-mlir
ref: refs/heads/main
submodules: 'true'

- name: Setup workspace
uses: ./.github/actions/setup-build

- name: Build Python wheels and smoke test
run: |
cd $GITHUB_WORKSPACE
tm_package_version="$(printf '%(%Y%m%d)T.${{ github.run_number }}')"
echo "tm_package_version=${tm_package_version}" >> $GITHUB_ENV
printf "TORCH_MLIR_PYTHON_PACKAGE_VERSION=%s\n" ${tm_package_version} > ./torch_mlir_package_version
TM_PYTHON_VERSIONS=${{ matrix.py_version }} TM_PACKAGES=${{ matrix.package }} ./build_tools/python_deploy/build_linux_packages.sh
- name: Make assets available in dist
run: |
mkdir dist
cp build_tools/python_deploy/wheelhouse/torch*.whl dist/
- name: Upload python wheels
uses: actions/upload-artifact@v4
with:
if-no-files-found: error
name: snapshot-${{ matrix.package }}-${{ matrix.py_version }}-${{ env.tm_package_version }}
path: dist

- name: Release python wheels
uses: ncipollo/release-action@v1.14.0
with:
artifacts: dist/*.whl
token: "${{ secrets.GITHUB_TOKEN }}"
tag: "dev-wheels"
name: "dev-wheels"
body: "Automatic snapshot release of torch-mlir python wheels."
removeArtifacts: false
allowUpdates: true
replacesArtifacts: true
makeLatest: true

0 comments on commit 3c6d32e

Please sign in to comment.