Remove xunit.console and make VSTest the default runner both locally and in CI

[ViktorHofer]

Today we support two different test runners: xunit.console and VSTest. The former is used when invoking the test target dotnet msbuild /t:Test and when running on CI. The latter is used when using VS Test Explorer, F5, the dotnet msbuild /t:Run target or when invoking dotnet test. To leverage the new crash dump / hang dump features that @davidfowl is driving in partnership with the VSTest team (cc @nohwnd), we should switch to VSTest and remove xunit.console entirely.

This issue only tracks the libraries side for non-mobile and non-WASM configurations. Mobile and WASM use a different harness called xharness which is out-of-scope at the moment.

1. Remove xunit.console:

• https://github.com/dotnet/runtime/blob/master/eng/testing/xunit/xunit.console.targets
• https://github.com/dotnet/runtime/blob/master/eng/testing/xunit/xunit.targets#L17

2. Remove runner templates for non-mobile and non-wasm:

• https://github.com/dotnet/runtime/blob/master/eng/testing/RunnerTemplate.cmd
• https://github.com/dotnet/runtime/blob/master/eng/testing/RunnerTemplate.sh
• 85bfbfc#diff-e7bc39410f7a119e7adb69b17006c452
• https://github.com/dotnet/runtime/blob/master/eng/testing/tests.targets#L22-L25

3. Revert 85bfbfc#diff-e7bc39410f7a119e7adb69b17006c452

4. Condition the RunTemplate and Test target logic on TargetsMobile and WASM:

• https://github.com/dotnet/runtime/blob/master/eng/testing/tests.targets#L109-L112
• https://github.com/dotnet/runtime/blob/master/eng/testing/tests.targets#L40-L104

5. Use dotnet test in CI instead of RunTests.cmd/sh for non-mobile and non-wasm:

• runtime/src/libraries/sendtohelix.proj

  Lines 69 to 70 in 132be64

  	<HelixCommand Condition="'$(TargetOS)' == 'Windows_NT'">call RunTests.cmd --runtime-path %HELIX_CORRELATION_PAYLOAD%</HelixCommand>
  	<HelixCommand Condition="'$(TargetOS)' != 'Windows_NT'">./RunTests.sh --runtime-path "$HELIX_CORRELATION_PAYLOAD"</HelixCommand>

6. Fix failing tests (ie Microsoft.Extensions.DependencyModel.Tests) and make sure that code coverage works.

7. Optional: use dotnet publish instead of the ArchiveTests target:

• https://github.com/dotnet/runtime/blob/master/eng/testing/tests.targets#L27-L38

cc @Anipik @ericstj @safern

[ghost]

Tagging subscribers to this area: @safern, @ViktorHofer
Notify danmosemsft if you want to be subscribed.

[ViktorHofer]

Blocked until 5.0 P8 SDK is released.

[tmds]

I'm interested in this because it will make long running tests produce a coredump.
On our rhel8 arm64 CI machine, occasionally we see some long running tests, and having these coredumps will help debug them.

[tmds]

@ViktorHofer is this something you'll be working on short term? I've been ignoring failures from long running tests on our arm64 rhel8 ci machine because having coredumps from vstest will make it easier to debug them.

[ViktorHofer]

@tmds #39923 is finally ready. It should be merged as soon as I discussed the new device bring-up scenario offline with my colleagues.

[PRIMETSS]

Spent the last few days trying to get v6 & v5 building locally to look at SerialPort Libs and kept having issue where build seemed to break with open files errors. Traced it to xunit.console.exe is not terminating if run dotnet build /t:test
Seems related to https://github.com/xunit/xunit/issues/1811 Cant even KILL the process, have to reboot!
Begs the question, how does one Test Libs LOCALLY?? Is everyone using the CI only? (guess it runs in Container and gets wiped out each build cycle so runs clean?) Will try JUST using VS to run tests, as this might work if you dont mix running tests in CLI.
Just baffled how this isn't a more common question? (me?)

[ViktorHofer]

Blocked by microsoft/vstest#3595