Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Solve CI vulnerability #1581

Merged
merged 47 commits into from
Jan 6, 2021
Merged

Solve CI vulnerability #1581

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
fcf2d44
test
yinweisu Dec 28, 2020
f7ceb9c
test
yinweisu Dec 28, 2020
ed69091
test
yinweisu Dec 28, 2020
4b47d54
test
yinweisu Dec 28, 2020
7f92d57
test
yinweisu Dec 30, 2020
b5f5a4e
test (#14)
yinweisu Dec 30, 2020
e4ad73b
test
yinweisu Dec 30, 2020
38bf4d9
test
yinweisu Dec 30, 2020
4c89c4f
test
yinweisu Dec 30, 2020
837afba
test
yinweisu Dec 30, 2020
cfa2807
test repo name
yinweisu Dec 30, 2020
72020e4
test
yinweisu Dec 30, 2020
b3e19b0
test
yinweisu Dec 30, 2020
8b47aad
test
yinweisu Dec 30, 2020
5c995b3
test
yinweisu Dec 30, 2020
7e5bce9
test
yinweisu Dec 30, 2020
680a39f
test
yinweisu Dec 30, 2020
a79bae8
test
yinweisu Dec 30, 2020
11fd7c5
test
yinweisu Dec 30, 2020
be42c9d
test
yinweisu Dec 30, 2020
d779238
test
yinweisu Dec 30, 2020
fe51122
test
yinweisu Dec 31, 2020
0b5f100
test
yinweisu Dec 31, 2020
973b1d3
test
yinweisu Dec 31, 2020
1fabb3a
test
yinweisu Dec 31, 2020
b798248
test
yinweisu Dec 31, 2020
23b1990
test
yinweisu Dec 31, 2020
cb1afd0
test
yinweisu Dec 31, 2020
a6748b4
test (#15)
yinweisu Dec 31, 2020
e355bb6
test (#16)
yinweisu Dec 31, 2020
31a2ce6
test (#17)
yinweisu Dec 31, 2020
8178767
test
yinweisu Dec 31, 2020
530c21d
test (#18)
yinweisu Dec 31, 2020
786e7e6
test (#19)
yinweisu Jan 4, 2021
6010565
test (#20)
yinweisu Jan 4, 2021
545eae0
fix
yinweisu Jan 4, 2021
0ce0cb0
fix
yinweisu Jan 4, 2021
85cdae1
fix
yinweisu Jan 4, 2021
9ef495e
full test
yinweisu Jan 4, 2021
9d3d249
fix
yinweisu Jan 4, 2021
6609c50
test pr (#21)
yinweisu Jan 5, 2021
cc00ac6
test pr (#22)
yinweisu Jan 5, 2021
305a408
test pr (#23)
yinweisu Jan 5, 2021
34dcbb1
full test (#24)
yinweisu Jan 6, 2021
c339e1c
finish testing
yinweisu Jan 6, 2021
7f0e43f
readme
yinweisu Jan 6, 2021
85ada88
trigger workflow
yinweisu Jan 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions .github/workflows/build_docs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ EFS=/mnt/efs
mkdir -p ~/.mxnet/datasets
for f in $EFS/.mxnet/datasets/*; do
if [ -d "$f" ]; then
# Will not run if no directories are available
# Will not run if no directories are available
ln -s $f ~/.mxnet/datasets/$(basename "$f")
fi
done
Expand All @@ -26,13 +26,10 @@ sed -i.bak 's/2196f3/178dc9/g' build/html/_static/sphinx_materialdesign_theme.cs
sed -i.bak 's/pre{padding:1rem;margin:1.5rem\\s0;overflow:auto;overflow-y:hidden}/pre{padding:1rem;margin:1.5rem 0;overflow:auto;overflow-y:scroll}/g' build/html/_static/sphinx_materialdesign_theme.css

if [[ ($BRANCH == master) && ($GIT_REPO == dmlc/gluon-cv) ]]; then
# aws s3 cp s3://gluon-cv.mxnet.io/coverage.svg build/html/coverage.svg
aws s3 sync --delete build/html/ s3://gluon-cv.mxnet.io/ --acl public-read --cache-control max-age=7200
# aws s3 cp build/html/coverage.svg s3://gluon-cv.mxnet.io/coverage.svg --acl public-read --cache-control max-age=300
echo "Uploaded doc to http://gluon-cv.mxnet.io"
echo master
else
# aws s3 cp s3://gluon-vision-staging/$PR_NUMBER/$COMMIT_SHA/coverage.svg build/html/coverage.svg
aws s3 sync --delete build/html/ s3://gluon-vision-staging/PR-$PR_NUMBER/$COMMIT_SHA/ --acl public-read
echo "Uploaded doc to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-$PR_NUMBER/$COMMIT_SHA/index.html"
echo $GIT_REPO: $BRANCH
Expand Down
283 changes: 283 additions & 0 deletions .github/workflows/build_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,283 @@
name: Build Test
on:
workflow_run:
workflows: ["Unit Test"]
types:
- completed
jobs:
unittest-check:
runs-on: ubuntu-latest
steps:
- uses: haya14busa/action-workflow_run-status@v1
- name: Fail if unittests failed
if: ${{ github.event.workflow_run.conclusion != 'success' }}
run: exit 1
model_zoo_mxnet:
needs: unittest-check
runs-on: ubuntu-latest
steps:
- uses: haya14busa/action-workflow_run-status@v1
- name: Log
env:
GITHUB_CONTEXT: ${{ toJson(github) }}
run: |
echo "$GITHUB_CONTEXT"
- name: Download artifact
uses: dawidd6/action-download-artifact@v2
with:
workflow: unittest.yml
run_id: ${{ github.event.workflow_run.id }}
- name: Get git context(For push)
if: ${{ github.event.workflow_run.event == 'push' }}
run: |
ref=$(head -n 1 artifact/ref.atf)
repo=$(head -n 1 artifact/repo.atf)
echo "REF=$ref" >> $GITHUB_ENV
echo "REPO=$repo" >> $GITHUB_ENV
- name: Get git context(For pull request)
if: ${{ github.event.workflow_run.event == 'pull_request' }}
run: |
ref=$(head -n 1 artifact/ref.atf)
repo=$(head -n 1 artifact/repo.atf)
pr=$(head -n 1 artifact/pr.atf)
echo "REF=$ref" >> $GITHUB_ENV
echo "REPO=$repo" >> $GITHUB_ENV
echo "PR=$pr" >> $GITHUB_ENV
- name: Checkout repository
uses: actions/checkout@v2
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.GLUONCV_DEV_ACCESS_ID }}
aws-secret-access-key: ${{ secrets.GLUONCV_DEV_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Install dependencies
run: |
pip install --upgrade --force-reinstall --no-deps .
pip install boto3
- name: Test model_zoo_mxnet on AWS Batch(For push)
shell: bash -l {0}
if: ${{ github.event.workflow_run.event == 'push' }}
run: |
echo "Start submitting job"
python ./tools/batch/submit-job.py --region us-east-1 \
--job-type ci \
--name GluonCV-GPU-ModelZooMxnet-${{ env.REF }} \
--source-ref ${{ env.REF }} \
--work-dir . \
--remote https://github.com/${{ env.REPO }} \
--command "chmod +x ./.github/workflows/gpu_test.sh && ./.github/workflows/gpu_test.sh gluoncv tests/model_zoo" \
--wait
- name: Test model_zoo_mxnet on AWS Batch(For pull request)
if: ${{ github.event.workflow_run.event == 'pull_request' }}
run: |
echo "Start submitting job"
python ./tools/batch/submit-job.py --region us-east-1 \
--job-type ci \
--name GluonCV-GPU-ModelZooMxnet-PR#${{ env.PR }} \
--source-ref ${{ env.REF }} \
--work-dir . \
--remote https://github.com/${{ env.REPO }} \
--command "chmod +x ./.github/workflows/gpu_test.sh && ./.github/workflows/gpu_test.sh gluoncv tests/model_zoo" \
--wait
model_zoo_torch:
needs: unittest-check
runs-on: ubuntu-latest
steps:
- uses: haya14busa/action-workflow_run-status@v1
- name: Download artifact
uses: dawidd6/action-download-artifact@v2
with:
workflow: unittest.yml
run_id: ${{ github.event.workflow_run.id }}
- name: Get git context(For push)
if: ${{ github.event.workflow_run.event == 'push' }}
run: |
ref=$(head -n 1 artifact/ref.atf)
repo=$(head -n 1 artifact/repo.atf)
echo "REF=$ref" >> $GITHUB_ENV
echo "REPO=$repo" >> $GITHUB_ENV
- name: Get git context(For pull request)
if: ${{ github.event.workflow_run.event == 'pull_request' }}
run: |
ref=$(head -n 1 artifact/ref.atf)
repo=$(head -n 1 artifact/repo.atf)
pr=$(head -n 1 artifact/pr.atf)
echo "REF=$ref" >> $GITHUB_ENV
echo "REPO=$repo" >> $GITHUB_ENV
echo "PR=$pr" >> $GITHUB_ENV
- name: Checkout repository
uses: actions/checkout@v2
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.GLUONCV_DEV_ACCESS_ID }}
aws-secret-access-key: ${{ secrets.GLUONCV_DEV_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Install dependencies
run: |
pip install --upgrade --force-reinstall --no-deps .
pip install boto3
- name: Test model_zoo_torch on AWS Batch(For push)
shell: bash -l {0}
if: ${{ github.event.workflow_run.event == 'push' }}
run: |
echo "Start submitting job"
python ./tools/batch/submit-job.py --region us-east-1 \
--job-type ci \
--name GluonCV-GPU-ModelZooTorch-${{ env.REF }} \
--source-ref ${{ env.REF }} \
--work-dir . \
--remote https://github.com/${{ env.REPO }} \
--command "chmod +x ./.github/workflows/gpu_test.sh && ./.github/workflows/gpu_test.sh gluoncv/torch tests/model_zoo_torch" \
--wait
- name: Test model_zoo_torch on AWS Batch(For pull request)
if: ${{ github.event.workflow_run.event == 'pull_request' }}
run: |
echo "Start submitting job"
python ./tools/batch/submit-job.py --region us-east-1 \
--job-type ci \
--name GluonCV-GPU-ModelZooTorch-PR#${{ env.PR }} \
--source-ref ${{ env.REF }} \
--work-dir . \
--remote https://github.com/${{ env.REPO }} \
--command "chmod +x ./.github/workflows/gpu_test.sh && ./.github/workflows/gpu_test.sh gluoncv/torch tests/model_zoo_torch" \
--wait
auto:
needs: unittest-check
runs-on: ubuntu-latest
steps:
- uses: haya14busa/action-workflow_run-status@v1
- name: Download artifact
uses: dawidd6/action-download-artifact@v2
with:
workflow: unittest.yml
run_id: ${{ github.event.workflow_run.id }}
- name: Get git context(For push)
if: ${{ github.event.workflow_run.event == 'push' }}
run: |
ref=$(head -n 1 artifact/ref.atf)
repo=$(head -n 1 artifact/repo.atf)
echo "REF=$ref" >> $GITHUB_ENV
echo "REPO=$repo" >> $GITHUB_ENV
- name: Get git context(For pull request)
if: ${{ github.event.workflow_run.event == 'pull_request' }}
run: |
ref=$(head -n 1 artifact/ref.atf)
repo=$(head -n 1 artifact/repo.atf)
pr=$(head -n 1 artifact/pr.atf)
echo "REF=$ref" >> $GITHUB_ENV
echo "REPO=$repo" >> $GITHUB_ENV
echo "PR=$pr" >> $GITHUB_ENV
- name: Checkout repository
uses: actions/checkout@v2
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.GLUONCV_DEV_ACCESS_ID }}
aws-secret-access-key: ${{ secrets.GLUONCV_DEV_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Install dependencies
run: |
pip install --upgrade --force-reinstall --no-deps .
pip install boto3
- name: Test model_zoo_torch on AWS Batch(For push)
shell: bash -l {0}
if: ${{ github.event.workflow_run.event == 'push' }}
run: |
echo "Start submitting job"
python ./tools/batch/submit-job.py --region us-east-1 \
--job-type ci \
--name GluonCV-GPU-Auto-${{ env.REF }} \
--source-ref ${{ env.REF }} \
--work-dir . \
--remote https://github.com/${{ env.REPO }} \
--command "chmod +x ./.github/workflows/gpu_test.sh && ./.github/workflows/gpu_test.sh gluoncv tests/auto" \
--wait
- name: Test model_zoo_torch on AWS Batch(For pull request)
if: ${{ github.event.workflow_run.event == 'pull_request' }}
run: |
echo "Start submitting job"
python ./tools/batch/submit-job.py --region us-east-1 \
--job-type ci \
--name GluonCV-GPU-Auto-PR#${{ env.PR }} \
--source-ref ${{ env.REF }} \
--work-dir . \
--remote https://github.com/${{ env.REPO }} \
--command "chmod +x ./.github/workflows/gpu_test.sh && ./.github/workflows/gpu_test.sh gluoncv tests/auto" \
--wait
build-docs:
needs: [unittest-check, model_zoo_mxnet, model_zoo_torch, auto]
runs-on: ubuntu-latest
steps:
- uses: haya14busa/action-workflow_run-status@v1
- name: Checkout repository
uses: actions/checkout@v2
- name: Download artifact
uses: dawidd6/action-download-artifact@v2
with:
workflow: unittest.yml
run_id: ${{ github.event.workflow_run.id }}
- name: Get git context(For push)
if: ${{ github.event.workflow_run.event == 'push' }}
run: |
ref=$(head -n 1 artifact/ref.atf)
repo=$(head -n 1 artifact/repo.atf)
echo "REF=$ref" >> $GITHUB_ENV
echo "REPO=$repo" >> $GITHUB_ENV
- name: Get git context(For pull request)
if: ${{ github.event.workflow_run.event == 'pull_request' }}
run: |
ref=$(head -n 1 artifact/ref.atf)
repo=$(head -n 1 artifact/repo.atf)
pr=$(head -n 1 artifact/pr.atf)
sha=$(head -n 1 artifact/sha.atf)
short_sha=$(git rev-parse --short "$sha")
echo "REF=$ref" >> $GITHUB_ENV
echo "REPO=$repo" >> $GITHUB_ENV
echo "PR=$pr" >> $GITHUB_ENV
echo "$SHORT_SHA"
echo "SHORT_SHA=$short_sha" >> $GITHUB_ENV
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.GLUONCV_DEV_ACCESS_ID }}
aws-secret-access-key: ${{ secrets.GLUONCV_DEV_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Install dependencies
run: |
pip install --upgrade --force-reinstall --no-deps .
pip install boto3
- name: Build docs on AWS Batch(For push)
shell: bash -l {0}
if: ${{ github.event.workflow_run.event == 'push' }}
run: |
echo "Start submitting job"
python ./tools/batch/submit-job.py --region us-east-1 \
--job-type ci \
--name GluonCV-GPU-BuildDocs-${{ env.REF }} \
--source-ref ${{ env.REF }} \
--work-dir . \
--remote https://github.com/${{ env.REPO }} \
--command "chmod +x ./.github/workflows/build_docs.sh && ./.github/workflows/build_docs.sh ${{ env.REF }} ${{ env.SHORT_SHA }} ${{ env.REPO }} ${{ env.PR }}" \
--wait
- name: Build docs on AWS Batch(For pull request)
if: ${{ github.event.workflow_run.event == 'pull_request' }}
run: |
echo "Start submitting job"
python ./tools/batch/submit-job.py --region us-east-1 \
--job-type ci \
--name GluonCV-GPU-BuildDocs-PR#${{ env.PR }} \
--source-ref ${{ env.REF }} \
--work-dir . \
--remote https://github.com/${{ env.REPO }} \
--command "chmod +x ./.github/workflows/build_docs.sh && ./.github/workflows/build_docs.sh ${{ env.REF }} ${{ env.SHORT_SHA }} ${{ env.REPO }} ${{ env.PR }} " \
--wait
- name: Comment on PR
if: ${{ github.event.workflow_run.event == 'pull_request' }}
uses: peter-evans/create-or-update-comment@v1.4.3
with:
issue-number: ${{ env.PR }}
body: |
Job PR-${{ env.PR }}-${{ env.SHORT_SHA }} is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-${{ env.PR }}/${{ env.SHORT_SHA }}/index.html
Loading