Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terminal logger integration #4843

Closed
5 of 6 tasks
nohwnd opened this issue Jan 22, 2024 · 6 comments
Closed
5 of 6 tasks

Terminal logger integration #4843

nohwnd opened this issue Jan 22, 2024 · 6 comments
Assignees

Comments

@nohwnd
Copy link
Member

nohwnd commented Jan 22, 2024

.NET 9 preview1 (since 9.0.100-alpha.1.24073.1) is shipping with MSBuild TerminalLogger enabled by default for most dotnet workloads, including dotnet test. The experience is not perfect, so I am looking at ways to improve it, together with the terminal logger team.

The change for the vstest logger to integrate better with MSBuild was done here #2702 , it traces errors via msbuild output so it is visible in binary logger viewer etc.

This experience, paired with disabling other workarounds such as not reusing test nodes is enabled by default in net9 and can be opt-out by -p:VSTestUseMSBuildOutput=false build property or by MSBUILDENSURESTDOUTFORTASKPROCESSES=1 (after #4853). The change is implemented here: dotnet/sdk#38098

My wishlist:

Temporary output (appears when it happens and is rolled out when there is too much):

  • getting passed tests as temporary output on the screen
  • tests can be colorized (a "bug" doubles the time, most likely this is not accounting for the ansi codes to go missing once rendered? )
    colorization-issue

Static output:

  • identification of the "task", so we can distinguish build success from test success
  • success of test assembly, similar how it is reported for successfully built project
  • single summary of test run with test counts
  • summary of attachments
@nohwnd
Copy link
Member Author

nohwnd commented Jan 30, 2024

Progress so far:

terminal-logger

What also cannot be seen here is that binlog is getting full summary of the failure, and full summary passed tests. As well as messages from outputs. E.g.
image

There is also one summary for the whole run, and not summary per project/tfm.

@nohwnd
Copy link
Member Author

nohwnd commented Jan 30, 2024

msbuild part: dotnet/msbuild@main...exp/terminal-logger-for-vstest
vstest part: nohwnd/vstest@main...terminal-logger-for-vstest

discussion points:

  • colorization of "targets" (described above)

  • how to evolve this (msbuild does not ship directly with fixed version of vstest, so we will have hard time changing the messages?)

  • better integration pattern ? All the update messages go to binlog where it is messy
    image

  • allowing the new Test target to use this as well (-p:UseMSBuildTestInfrastructure=true)

  • allowing MSTest runner to use this as well - we use generic target and generic names for the messages. should be good enough for now.

  • detecting and disabling TerminalLogger mode, so user does not get spammed with all passed tests

  • optimizations (I am for sure committing a lot of sins against string parsing best practice)

@nohwnd
Copy link
Member Author

nohwnd commented Feb 6, 2024

Dispatch to inner needs to somehow parallelize
image

Solution proposed here, works great. Just needs the Property rename, because disabling it won't fully disable test parallelization. E.g. -p:TestTfmInParallel=false

@nohwnd
Copy link
Member Author

nohwnd commented Mar 22, 2024

Next up:

  • Skipped! when all tests are skipped, rather than passed.
  • support for multiline error output, which is soon coming to msbuild
  • attachments in summary
  • fix how trx logger outputs the file path, to be picked up be attachments
  • ROLL BACK: roll back the passed and skipped tests, while we wait for proper progress api in msbuild

@nohwnd
Copy link
Member Author

nohwnd commented Jul 8, 2024

We ended up writing most of the output in net9-preview6, making it as useful as before.

@nohwnd nohwnd closed this as completed Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant