Merge release 1.4.0 to master

Release 1.4.0 to master The Ginkgo team is proud to announce the new Ginkgo minor release 1.4.0. This release brings most of the Ginkgo functionality to the Intel DPC++ ecosystem which enables Intel-GPU and CPU execution. The only Ginkgo features which have not been ported yet are some preconditioners. Ginkgo's mixed-precision support is greatly enhanced thanks to: 1. The new Accessor concept, which allows writing kernels featuring on-the-fly memory compression, among other features. The accessor can be used as header-only, see the [accessor BLAS benchmarks repository](https://github.com/ginkgo-project/accessor-BLAS/tree/develop) as a usage example. 2. All LinOps now transparently support mixed-precision execution. By default, this is done through a temporary copy which may have a performance impact but already allows mixed-precision research. Native mixed-precision ELL kernels are implemented which do not see this cost. The accessor is also leveraged in a new CB-GMRES solver which allows for performance improvements by compressing the Krylov basis vectors. Many other features have been added to Ginkgo, such as reordering support, a new IDR solver, Incomplete Cholesky preconditioner, matrix assembly support (only CPU for now), machine topology information, and more! Supported systems and requirements: + For all platforms, cmake 3.13+ + C++14 compliant compiler + Linux and MacOS + gcc: 5.3+, 6.3+, 7.3+, all versions after 8.1+ + clang: 3.9+ + Intel compiler: 2018+ + Apple LLVM: 8.0+ + CUDA module: CUDA 9.0+ + HIP module: ROCm 3.5+ + DPC++ module: Intel OneAPI 2021.3. Set the CXX compiler to `dpcpp`. + Windows + MinGW and Cygwin: gcc 5.3+, 6.3+, 7.3+, all versions after 8.1+ + Microsoft Visual Studio: VS 2019 + CUDA module: CUDA 9.0+, Microsoft Visual Studio + OpenMP module: MinGW or Cygwin. Algorithm and important feature additions: + Add a new DPC++ Executor for SYCL execution and other base utilities [#648](#648), [#661](#661), [#757](#757), [#832](#832) + Port matrix formats, solvers and related kernels to DPC++. For some kernels, also make use of a shared kernel implementation for all executors (except Reference). [#710](#710), [#799](#799), [#779](#779), [#733](#733), [#844](#844), [#843](#843), [#789](#789), [#845](#845), [#849](#849), [#855](#855), [#856](#856) + Add accessors which allow multi-precision kernels, among other things. [#643](#643), [#708](#708) + Add support for mixed precision operations through apply in all LinOps. [#677](#677) + Add incomplete Cholesky factorizations and preconditioners as well as some improvements to ILU. [#672](#672), [#837](#837), [#846](#846) + Add an AMGX implementation and kernels on all devices but DPC++. [#528](#528), [#695](#695), [#860](#860) + Add a new mixed-precision capability solver, Compressed Basis GMRES (CB-GMRES). [#693](#693), [#763](#763) + Add the IDR(s) solver. [#620](#620) + Add a new fixed-size block CSR matrix format (for the Reference executor). [#671](#671), [#730](#730) + Add native mixed-precision support to the ELL format. [#717](#717), [#780](#780) + Add Reverse Cuthill-McKee reordering [#500](#500), [#649](#649) + Add matrix assembly support on CPUs. [#644](#644) + Extends ISAI from triangular to general and spd matrices. [#690](#690) Other additions: + Add the possibility to apply real matrices to complex vectors. [#655](#655), [#658](#658) + Add functions to compute the absolute of a matrix format. [#636](#636) + Add symmetric permutation and improve existing permutations. [#684](#684), [#657](#657), [#663](#663) + Add a MachineTopology class with HWLOC support [#554](#554), [#697](#697) + Add an implicit residual norm criterion. [#702](#702), [#818](#818), [#850](#850) + Row-major accessor is generalized to more than 2 dimensions and a new "block column-major" accessor has been added. [#707](#707) + Add an heat equation example. [#698](#698), [#706](#706) + Add ccache support in CMake and CI. [#725](#725), [#739](#739) + Allow tuning and benchmarking variables non intrusively. [#692](#692) + Add triangular solver benchmark [#664](#664) + Add benchmarks for BLAS operations [#772](#772), [#829](#829) + Add support for different precisions and consistent index types in benchmarks. [#675](#675), [#828](#828) + Add a Github bot system to facilitate development and PR management. [#667](#667), [#674](#674), [#689](#689), [#853](#853) + Add Intel (DPC++) CI support and enable CI on HPC systems. [#736](#736), [#751](#751), [#781](#781) + Add ssh debugging for Github Actions CI. [#749](#749) + Add pipeline segmentation for better CI speed. [#737](#737) Changes: + Add a Scalar Jacobi specialization and kernels. [#808](#808), [#834](#834), [#854](#854) + Add implicit residual log for solvers and benchmarks. [#714](#714) + Change handling of the conjugate in the dense dot product. [#755](#755) + Improved Dense stride handling. [#774](#774) + Multiple improvements to the OpenMP kernels performance, including COO, an exclusive prefix sum, and more. [#703](#703), [#765](#765), [#740](#740) + Allow specialization of submatrix and other dense creation functions in solvers. [#718](#718) + Improved Identity constructor and treatment of rectangular matrices. [#646](#646) + Allow CUDA/HIP executors to select allocation mode. [#758](#758) + Check if executors share the same memory. [#670](#670) + Improve test install and smoke testing support. [#721](#721) + Update the JOSS paper citation and add publications in the documentation. [#629](#629), [#724](#724) + Improve the version output. [#806](#806) + Add some utilities for dim and span. [#821](#821) + Improved solver and preconditioner benchmarks. [#660](#660) + Improve benchmark timing and output. [#669](#669), [#791](#791), [#801](#801), [#812](#812) Fixes: + Sorting fix for the Jacobi preconditioner. [#659](#659) + Also log the first residual norm in CGS [#735](#735) + Fix BiCG and HIP CSR to work with complex matrices. [#651](#651) + Fix Coo SpMV on strided vectors. [#807](#807) + Fix segfault of extract_diagonal, add short-and-fat test. [#769](#769) + Fix device_reset issue by moving counter/mutex to device. [#810](#810) + Fix `EnableLogging` superclass. [#841](#841) + Support ROCm 4.1.x and breaking HIP_PLATFORM changes. [#726](#726) + Decreased test size for a few device tests. [#742](#742) + Fix multiple issues with our CMake HIP and RPATH setup. [#712](#712), [#745](#745), [#709](#709) + Cleanup our CMake installation step. [#713](#713) + Various simplification and fixes to the Windows CMake setup. [#720](#720), [#785](#785) + Simplify third-party integration. [#786](#786) + Improve Ginkgo device arch flags management. [#696](#696) + Other fixes and improvements to the CMake setup. [#685](#685), [#792](#792), [#705](#705), [#836](#836) + Clarification of dense norm documentation [#784](#784) + Various development tools fixes and improvements [#738](#738), [#830](#830), [#840](#840) + Make multiple operators/constructors explicit. [#650](#650), [#761](#761) + Fix some issues, memory leaks and warnings found by MSVC. [#666](#666), [#731](#731) + Improved solver memory estimates and consistent iteration counts [#691](#691) + Various logger improvements and fixes [#728](#728), [#743](#743), [#754](#754) + Fix for ForwardIterator requirements in iterator_factory. [#665](#665) + Various benchmark fixes. [#647](#647), [#673](#673), [#722](#722) + Various CI fixes and improvements. [#642](#642), [#641](#641), [#795](#795), [#783](#783), [#793](#793), [#852](#852) Related PR: #866
ginkgo-project · Aug 23, 2021 · f811917 · f811917
2 parents 4678668 + 555f65e
commit f811917
Show file tree

Hide file tree

Showing 1,163 changed files with 256,370 additions and 17,426 deletions.
diff --git a/.github/bot-base.sh b/.github/bot-base.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+
+set -e
+
+API_HEADER="Accept: application/vnd.github.v3+json"
+AUTH_HEADER="Authorization: token $GITHUB_TOKEN"
+
+api_get() {
+  curl -X GET -s -H "${AUTH_HEADER}" -H "${API_HEADER}" "$1"
+}
+
+api_post() {
+  curl -X POST -s -H "${AUTH_HEADER}" -H "${API_HEADER}" "$1" -d "$2"
+}
+
+api_patch() {
+  curl -X PATCH -s -H "${AUTH_HEADER}" -H "${API_HEADER}" "$1" -d "$2"
+}
+
+api_delete() {
+  curl -X DELETE -s -H "${AUTH_HEADER}" -H "${API_HEADER}" "$1"
+}
diff --git a/.github/bot-pr-base.sh b/.github/bot-pr-base.sh
@@ -0,0 +1,100 @@
+#!/bin/bash
+
+source .github/bot-base.sh
+
+echo -n "Collecting information on triggering PR"
+PR_URL=$(jq -r .pull_request.url "$GITHUB_EVENT_PATH")
+if [[ "$PR_URL" == "null" ]]; then
+  # if this was triggered by an issue comment: get PR and commenter
+  echo -n .............
+  PR_URL=$(jq -er .issue.pull_request.url "$GITHUB_EVENT_PATH")
+  echo -n .
+  USER_LOGIN=$(jq -er ".comment.user.login" "$GITHUB_EVENT_PATH")
+  echo -n .
+  USER_URL=$(jq -er ".comment.user.url" "$GITHUB_EVENT_PATH")
+  echo -n .
+else
+  # else it was triggered by a PR sync: get PR creator
+  USER_LOGIN=$(jq -er ".pull_request.user.login" "$GITHUB_EVENT_PATH")
+  echo -n .
+  USER_URL=$(jq -er ".pull_request.user.url" "$GITHUB_EVENT_PATH")
+  echo -n .
+fi
+echo -n .
+PR_JSON=$(api_get $PR_URL)
+echo -n .
+PR_MERGED=$(echo "$PR_JSON" | jq -r .merged)
+echo -n .
+ISSUE_URL=$(echo "$PR_JSON" | jq -er ".issue_url")
+echo -n .
+BASE_REPO=$(echo "$PR_JSON" | jq -er .base.repo.full_name)
+echo -n .
+BASE_BRANCH=$(echo "$PR_JSON" | jq -er .base.ref)
+echo -n .
+HEAD_REPO=$(echo "$PR_JSON" | jq -er .head.repo.full_name)
+echo -n .
+HEAD_BRANCH=$(echo "$PR_JSON" | jq -er .head.ref)
+echo .
+
+BASE_URL="https://${GITHUB_ACTOR}:${GITHUB_TOKEN}@github.com/$BASE_REPO"
+HEAD_URL="https://${GITHUB_ACTOR}:${GITHUB_TOKEN}@github.com/$HEAD_REPO"
+
+JOB_URL="https://github.com/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"
+
+bot_delete_comments_matching() {
+  local search_matching="$1"
+  COMMENTS=$(api_get "$ISSUE_URL/comments" | jq -r '.[] | select((.user.login == "ginkgo-bot") and (.body | startswith('"\"$search_matching\""'))) | .url')
+  for URL in $COMMENTS; do
+    api_delete "$URL" > /dev/null
+  done
+}
+
+bot_comment() {
+  api_post "$ISSUE_URL/comments" "{\"body\":\"$1\"}" > /dev/null
+}
+
+bot_error() {
+  echo "$1"
+  bot_comment "Error: $1"
+  exit 1
+}
+
+bot_get_all_changed_files() {
+  local pr_url="$1"
+  local pr_files=""
+  local page="1"
+  while true; do
+    # this api allows 100 items per page
+    # github action uses `bash -e`. The last empty page will leads jq error, use `|| :` to ignore the error.
+    local pr_page_files=$(api_get "$pr_url/files?&per_page=100&page=${page}" | jq -er '.[] | select(.status != "removed") | .filename' || :)
+    if [ "${pr_page_files}" = "" ]; then
+      break
+    fi
+    if [ ! "${pr_files}" = "" ]; then
+      # add the same new line format as jq output
+      pr_files="${pr_files}"$'\n'
+    fi
+    pr_files="${pr_files}${pr_page_files}"
+    page=$(( page + 1 ))
+  done
+  echo "${pr_files}"
+}
+
+# collect info on the user that invoked the bot
+echo -n "Collecting information on triggering user"
+USER_JSON=$(api_get $USER_URL)
+echo .
+
+USER_NAME=$(echo "$USER_JSON" | jq -r ".name")
+if [[ "$USER_NAME" == "null" ]]; then
+	USER_NAME=$USER_LOGIN
+fi
+USER_EMAIL=$(echo "$USER_JSON" | jq -r ".email")
+if [[ "$USER_EMAIL" == "null" ]]; then
+	USER_EMAIL="$USER_LOGIN@users.noreply.github.com"
+fi
+USER_COMBINED="$USER_NAME <$USER_EMAIL>"
+
+if [[ "$PR_MERGED" == "true" ]]; then
+  bot_error "PR already merged!"
+fi
diff --git a/.github/bot-pr-format-base.sh b/.github/bot-pr-format-base.sh
@@ -0,0 +1,44 @@
+#!/bin/bash
+
+source .github/bot-pr-base.sh
+
+EXTENSION_REGEX='\.(cuh?|hpp|hpp\.inc?|cpp)$'
+FORMAT_HEADER_REGEX='^(benchmark|core|cuda|hip|include/ginkgo/core|omp|reference|dpcpp)/'
+FORMAT_REGEX='^(common|examples|test)/'
+
+echo "Retrieving PR file list"
+PR_FILES=$(bot_get_all_changed_files ${PR_URL})
+NUM=$(echo "${PR_FILES}" | wc -l)
+echo "PR has ${NUM} changed files"
+
+TO_FORMAT="$(echo "$PR_FILES" | grep -E $EXTENSION_REGEX || true)"
+
+git remote add fork "$HEAD_URL"
+git fetch fork "$HEAD_BRANCH"
+
+git config user.email "ginkgo.library@gmail.com"
+git config user.name "ginkgo-bot"
+
+# save scripts from develop
+pushd dev_tools/scripts
+cp add_license.sh format_header.sh update_ginkgo_header.sh /tmp
+popd
+
+# checkout current PR head
+LOCAL_BRANCH=format-tmp-$HEAD_BRANCH
+git checkout -b $LOCAL_BRANCH fork/$HEAD_BRANCH
+
+# restore files from develop
+cp /tmp/add_license.sh dev_tools/scripts/
+cp /tmp/format_header.sh dev_tools/scripts/
+cp /tmp/update_ginkgo_header.sh dev_tools/scripts/
+
+# format files
+CLANG_FORMAT=clang-format-9
+dev_tools/scripts/add_license.sh
+dev_tools/scripts/update_ginkgo_header.sh
+for f in $(echo "$TO_FORMAT" | grep -E $FORMAT_HEADER_REGEX); do dev_tools/scripts/format_header.sh "$f"; done
+for f in $(echo "$TO_FORMAT" | grep -E $FORMAT_REGEX); do "$CLANG_FORMAT" -i -style=file "$f"; done
+
+# restore formatting scripts so they don't appear in the diff
+git checkout -- dev_tools/scripts/*.sh
diff --git a/.github/check-format.sh b/.github/check-format.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+cp .github/bot-pr-format-base.sh /tmp
+source /tmp/bot-pr-format-base.sh
+
+# check for changed files, replace newlines by \n
+LIST_FILES=$(git diff --name-only | sed '$!s/$/\\n/' | tr -d '\n')
+
+git diff > /tmp/format.patch
+mv /tmp/format.patch .
+
+bot_delete_comments_matching "Error: The following files need to be formatted"
+
+if [[ "$LIST_FILES" != "" ]]; then
+  MESSAGE="The following files need to be formatted:\n"'```'"\n$LIST_FILES\n"'```'
+  MESSAGE="$MESSAGE\nYou can find a formatting patch under **Artifacts** [here]"
+  MESSAGE="$MESSAGE($JOB_URL) or run "'`format!` if you have write access to Ginkgo'
+  bot_error "$MESSAGE"
+fi
diff --git a/.github/format.sh b/.github/format.sh
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+cp .github/bot-pr-format-base.sh /tmp
+source /tmp/bot-pr-format-base.sh
+
+# check for changed files, replace newlines by \n
+LIST_FILES=$(git diff --name-only | sed '$!s/$/\\n/' | tr -d '\n')
+
+# commit changes if necessary
+if [[ "$LIST_FILES" != "" ]]; then
+  git commit -a -m "Format files
+
+Co-authored-by: $USER_COMBINED"
+  git push fork $LOCAL_BRANCH:$HEAD_BRANCH 2>&1 || bot_error "Cannot push formatted branch, are edits for maintainers allowed?"
+fi
diff --git a/.github/label.sh b/.github/label.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+
+source .github/bot-pr-base.sh
+
+echo "Retrieving PR file list"
+PR_FILES=$(bot_get_all_changed_files ${PR_URL})
+NUM=$(echo "${PR_FILES}" | wc -l)
+echo "PR has ${NUM} changed files"
+
+echo "Retrieving PR label list"
+OLD_LABELS=$(api_get "$ISSUE_URL" | jq -er '[.labels | .[] | .name]')
+
+
+label_match() {
+  if echo "$PR_FILES" | grep -qE "$2"; then
+    echo "+[\"$1\"]"
+  fi
+}
+
+LABELS="[]"
+LABELS=$LABELS$(label_match mod:core '(^core/|^include/)')
+LABELS=$LABELS$(label_match mod:reference '^reference/')
+LABELS=$LABELS$(label_match mod:openmp '^omp/')
+LABELS=$LABELS$(label_match mod:cuda '(^cuda/|^common/)')
+LABELS=$LABELS$(label_match mod:hip '(^hip/|^common/)')
+LABELS=$LABELS$(label_match mod:dpcpp '^dpcpp/')
+LABELS=$LABELS$(label_match reg:benchmarking '^benchmark/')
+LABELS=$LABELS$(label_match reg:example '^examples/')
+LABELS=$LABELS$(label_match reg:build '(cm|CM)ake')
+LABELS=$LABELS$(label_match reg:ci-cd '(^\.github/|\.yml$)')
+LABELS=$LABELS$(label_match reg:documentation '^doc/|\.md$')
+LABELS=$LABELS$(label_match reg:testing /test/)
+LABELS=$LABELS$(label_match reg:helper-scripts '^dev_tools/')
+LABELS=$LABELS$(label_match type:factorization /factorization/)
+LABELS=$LABELS$(label_match type:matrix-format /matrix/)
+LABELS=$LABELS$(label_match type:multigrid /multigrid/)
+LABELS=$LABELS$(label_match type:preconditioner /preconditioner/)
+LABELS=$LABELS$(label_match type:reordering /reorder/)
+LABELS=$LABELS$(label_match type:solver /solver/)
+LABELS=$LABELS$(label_match type:stopping-criteria /stop/)
+
+# if all mod: labels present: replace by mod:all
+LABELS=$(echo "$LABELS" | sed 's/.*mod:.*mod:.*mod:.*mod:.*mod:.*mod:[^"]*"\]/[]+["mod:all"]/')
+
+PATCH_BODY=$(jq -rn "{labels:($OLD_LABELS + $LABELS | unique)}")
+api_patch "$ISSUE_URL" "$PATCH_BODY" > /dev/null
diff --git a/.github/rebase.sh b/.github/rebase.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+source .github/bot-pr-base.sh
+
+git remote add base "$BASE_URL"
+git remote add fork "$HEAD_URL"
+
+git fetch base $BASE_BRANCH
+git fetch fork $HEAD_BRANCH
+
+git config user.email "$USER_EMAIL"
+git config user.name "$USER_NAME"
+
+LOCAL_BRANCH=rebase-tmp-$HEAD_BRANCH
+git checkout -b $LOCAL_BRANCH fork/$HEAD_BRANCH
+
+bot_delete_comments_matching "Error: Rebase failed"
+
+# do the rebase
+git rebase base/$BASE_BRANCH 2>&1 || bot_error "Rebase failed, see the related [Action]($JOB_URL) for details"
+
+# push back
+git push --force-with-lease fork $LOCAL_BRANCH:$HEAD_BRANCH 2>&1 || bot_error "Cannot push rebased branch, are edits for maintainers allowed?"
diff --git a/.github/workflows/bot-pr-comment.yml b/.github/workflows/bot-pr-comment.yml
@@ -0,0 +1,65 @@
+on:
+  issue_comment:
+    types: [created]
+name: OnCommentPR
+jobs:
+  label:
+    runs-on: ubuntu-latest
+    if: github.event.issue.pull_request != '' && github.event.comment.body == 'label!' && (github.event.comment.author_association == 'COLLABORATOR' || github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'OWNER')
+    steps:
+    - name: Checkout the latest code (shallow clone)
+      uses: actions/checkout@v2
+      with:
+        ref: develop
+    - name: Add appropriate labels
+      env:
+        GITHUB_TOKEN: ${{ secrets.BOT_TOKEN }}
+      run: cp .github/label.sh /tmp && /tmp/label.sh
+  check_format:
+    name: check-format
+    runs-on: ubuntu-18.04
+    if: github.event.issue.pull_request != '' && github.event.comment.body == 'check-format!' && (github.event.comment.author_association == 'COLLABORATOR' || github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'OWNER')
+    steps:
+    - name: Checkout the latest code (shallow clone)
+      uses: actions/checkout@v2
+      with:
+        ref: develop
+    - name: Check for formatting changes
+      env:
+        GITHUB_TOKEN: ${{ secrets.BOT_TOKEN }}
+      run: cp .github/check-format.sh /tmp && /tmp/check-format.sh
+    - name: Upload code formatting patch
+      if: failure()
+      uses: actions/upload-artifact@v2
+      with:
+          name: patch
+          path: format.patch
+  format:
+    name: format
+    runs-on: ubuntu-18.04
+    if: github.event.issue.pull_request != '' && github.event.comment.body == 'format!' && (github.event.comment.author_association == 'COLLABORATOR' || github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'OWNER')
+    steps:
+    - name: Checkout the latest code (shallow clone)
+      uses: actions/checkout@v2
+      with:
+        ref: develop
+        persist-credentials: false
+    - name: Commit formatting changes
+      env:
+        GITHUB_TOKEN: ${{ secrets.BOT_TOKEN }}
+      run: cp .github/format.sh /tmp && /tmp/format.sh
+  rebase:
+    name: rebase
+    if: github.event.issue.pull_request != '' && github.event.comment.body == 'rebase!' && (github.event.comment.author_association == 'COLLABORATOR' || github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'OWNER')
+    runs-on: ubuntu-latest
+    steps:
+    - name: Checkout the latest code
+      uses: actions/checkout@v2
+      with:
+        ref: develop
+        fetch-depth: 0
+        persist-credentials: false
+    - name: Automatic Rebase
+      env:
+        GITHUB_TOKEN: ${{ secrets.BOT_TOKEN }}
+      run: cp .github/rebase.sh /tmp && /tmp/rebase.sh
diff --git a/.github/workflows/bot-pr-created.yml b/.github/workflows/bot-pr-created.yml
@@ -0,0 +1,18 @@
+on:
+  pull_request_target:
+    types: [opened]
+
+name: OnNewPR
+jobs:
+  label:
+    runs-on: ubuntu-latest
+    if: github.event.pull_request.author_association == 'COLLABORATOR' || github.event.pull_request.author_association == 'MEMBER' || github.event.pull_request.author_association == 'OWNER'
+    steps:
+      - name: Checkout the latest code (shallow clone)
+        uses: actions/checkout@v2
+        with:
+          ref: develop
+      - name: Add appropriate labels
+        env:
+          GITHUB_TOKEN: ${{ secrets.BOT_TOKEN }}
+        run: .github/label.sh
diff --git a/.github/workflows/bot-pr-updated.yml b/.github/workflows/bot-pr-updated.yml
@@ -0,0 +1,24 @@
+on:
+  pull_request_target:
+    types: [opened,synchronize]
+
+name: OnSyncPR
+jobs:
+  check-format:
+    runs-on: ubuntu-18.04
+    if: github.event.pull_request.author_association == 'COLLABORATOR' || github.event.pull_request.author_association == 'MEMBER' || github.event.pull_request.author_association == 'OWNER'
+    steps:
+      - name: Checkout the latest code (shallow clone)
+        uses: actions/checkout@v2
+        with:
+          ref: develop
+      - name: Check for formatting changes
+        env:
+          GITHUB_TOKEN: ${{ secrets.BOT_TOKEN }}
+        run: cp .github/check-format.sh /tmp && /tmp/check-format.sh
+      - name: Upload code formatting patch
+        if: failure()
+        uses: actions/upload-artifact@v2
+        with:
+          name: patch
+          path: format.patch
diff --git a/.github/workflows/joss.yml b/.github/workflows/joss.yml
@@ -11,7 +11,9 @@ jobs:
     runs-on: [ubuntu-latest] 
 
     steps:
-    - uses: actions/checkout@v2
+    - name: Checkout the latest code (shallow clone)
+      uses: actions/checkout@v2
+
     - name: setup
       run: sudo apt-get install texlive-xetex pandoc pandoc-citeproc
     - name: info