Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As we've been adding more tests, the CI testing takes longer to run. The timing varies a lot, but to begin with it would take ~15-20 minutes, while recently with more tests it's taking ~20-30 minutes. This PR speeds things up a bit by submitting each
test_*.py
file as a separate CI job, which can be assigned to different workers. Previously we just ran all pytests in one job, so each test would run sequentially. As we continue to develop more tests this should help avoid the CI really dragging.Here the pipeline is generated dynamically: the main
.gitlab-ci.yml
job runs thegenerate_ci_yaml.py
script, which creates a.yml
file defining all the jobs from the available test files. That's uploaded as an artifact and used to trigger the next stage. So to add a new test file you can still just add thetest_something.py
file to the pytests directory and it will be picked up by the CI.Here's a view of it running, with some jobs finished and some still going:
The timing seems to now be in the 10-15 minute range, pretty much limited to the time it takes to run the slowest test file - usually the MNIST CNN one.