[ci] Automatically retry failed MSBuild/emulator tests. #7997
Conversation
jpobst
force-pushed
the
retry-failed-tests
branch
5 times, most recently
from
55aef28
to
0d8b820
Compare
| @@ -2,22 +2,55 @@ parameters: | |||
| testAssembly: # NUnit test assembly to run | |||
| testFilter: # Filter used to select tests (NUnit test selection language, not dotnet test filter language) | |||
| testRunTitle: # Title of the test run | |||
| testResultsTitle: # Title used to construct test results file name | |||
| retryFailedTests: true # Retry failed tests once | |||
There was a problem hiding this comment.
General thought but might it be better to add the [Retry] attribute to tests that we see failing somewhat regularly? NUnit does provide some built in support for this that we are already using in a handful of places. This would also give us a better understanding of the tests that do fail regularly so that we could try to improve them at some point.
There was a problem hiding this comment.
While I agree we should be trying to improve our test reliability, it does not seem like something we have been dedicating time to or are likely to start dedicating time to.
We have this data today: https://devdiv.visualstudio.com/DevDiv/_test/analytics?definitionId=11410&contextType=build, but it doesn't appear to be localized to a few tests. (550 unique tests failed over the past 14 days.) However, this may point to an issue with the suite(s) themselves that could be investigated.
It is good to point out that this data will not be available after this PR (unless the test fails on retry), as we will no longer be reporting initial failures to AzDO so that we have a green build. If we would like to continue having this data, a good compromise might be to perform automatic retries on PR builds but not for main builds.
I feel like the large number of unique test failures makes [Retry] a poor option. I looked at applying it at the class or assembly level, but that isn't supported. It can only be placed on individual tests.
Grendel and I also discovered that [Retry] is not available in NUnitLite which is used by the on-device tests.
There was a problem hiding this comment.
Breaking this down a bit further you can sort by test file, and you can see the vast majority are coming from Designer tests (though I'm not sure what branch).
Other than that, we clearly have a few core culprits under a ~90% pass rate:
Maybe it would be best to only enable this for the emulator test jobs for now, as that is where most issues seem to be? This seems like a good interim solution, at least until we can find a machine pool that will provide a more stable emulator/device for us.
I believe most of our test related CI headaches stem from using an emulator in nested virtualization, and could be resolved by investigating different test execution environments.
There was a problem hiding this comment.
👍 Set retryFailedTests: false for the regular MSBuild job, so it will only run on emulator tests.
There was a problem hiding this comment.
I believe most of our test related CI headaches stem from using an emulator in nested virtualization, and could be resolved by investigating different test execution environments.
I also wonder if we've ever considered we could have a bug in our code but we just blame the emulator rather than investigating. 🤔
| retry ` | ||
| --trx="$(Agent.TempDirectory)" ` | ||
| --outfile="${{ parameters.testAssembly }}.runsettings" | ||
| displayName: Look for failed tests |
There was a problem hiding this comment.
Do you have a link to the logic for this retry command for reference?
There was a problem hiding this comment.
* main: Bump to dotnet/installer@0ce891843a 8.0.100-preview.5.23228.7 (dotnet#7994) [ci] Automatically retry failed emulator tests. (dotnet#7997) [xaprepare] Combine 'AndroidTestDependencies' and 'EmulatorTestDependencies' scenarios. (dotnet#8006) [xaprepare] Provision 'platform-33_r02' (dotnet#8004) [Xamarin.Android.Build.Tasks] Add AndroidEnableRestrictToAttributes (dotnet#7990) [ci] Remove plots-to-appinsights. (dotnet#8002) [Xamarin.Android.Build.Tasks] remove NuGet dependencies (dotnet#8000) Bump to 33.0.56 $(AndroidNet7Version) (dotnet#7998) $(AndroidPackVersionSuffix)=preview.5; net8 is 34.0.0-preview.5 (dotnet#7996)
Previously, we have used
retryCountOnTaskFailureto retry some of our flaky test suites. However this is not a good solution for our MSBuild and emulator tests for 2 reasons:We need a solution that only retries the specific test(s) that failed. Unfortunately nothing like that exists, so we'll once again have to roll our own solution, and we can reuse
dotnet-test-slicerfor this.This involves some trickery to make everything show up correctly in the AzDO UI:
SucceededWithIssuesstatedotnet-test-slicer retry.runsettingsfile that only contains the failed tests sodotnet testcan run only themdotnet, and any test failures can be automatically reported to AzDOAffected Test Suites: