Test failure error message is unreadable in Azure DevOps #1664
Comments
Dotnet-GitSync-Bot
added
the
untriaged
|
@BruceForstall can you link to a build / run that has this issue? |
|
I think it's any test failure. E.g., It looks like the Linux/Mac jobs have |
|
/cc @ViktorHofer |
|
To make it worse, the AzDO UI for viewing ends up creating a very wide window with a horizontal scrollbar, so you have to constantly scroll back-and-forth (and read through a mass of no-whitespace text) to find the relevant bits of information. |
|
I always copy the text out and replace escaped newlines with newlines in np++. I would love to see this fixed. |
I've complained to AzDO about this before (we have same issue in Roslyn). They constantly push back the issue is on our end not there's but even when digging in I could never see where we were doing extra escaping. |
|
Is it possible that it's the Helix xunit reporter that's doing the escaping? That's the common factor between Roslyn and dotnet/runtime that other repos using xunit on AzDO don't have. |
@MattGal Can you suggest someone with knowledge of the Helix xunit reporter upload to AzDO that might be able to investigate this? |
Roslyn doesn't use Helix yet so it's not a common factor for this. Note: it should be possible to see if we are or aren't escaping by looking at the raw xUnit logs. They should be available in XML form for every test that fails in Helix correct? If so we can just look there and see if the values are escaped. That would at least eliminate infra in dotnet/runtime as the culprit. |
@jaredpar 's comment likely obviates this but you can read the full sources here: https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.Helix/Sdk/tools/azure-pipelines/reporter/azure_devops_result_publisher.py and @alexperovich wrote it. |
|
The build linked in the issue is too old for the results file to still exist. I looked at the latest build but the xunit file isn't avaliable for the JIT.HardwareIntrinsics work item. If we can catch another instance I can grab the xunit file and check. |
|
That workitem doesn't have the test results file uploaded for some reason. The older one did, it just doesn't exist anymore. |
|
@alexperovich What should I look for to find the uploaded test results file? Is there logging in this case that indicates why there is no uploaded test results file? |
|
I just used the JobId and WorkItemId in the Comments section in this api https://helix.dot.net/swagger/ui/index#!/WorkItem/WorkItem_ListFiles I believe the xunit file is only avaliable in that old run because you guys have the legacy xunit reporter turned on. The VSTS reporter doesn't actually upload the file. |
|
@alexperovich When I look at the Helix api above I can grab the console log for my example case, here: This has some interesting sections, e.g. uploading the results: Isn't this the Helix uploader? I guess you wanted to see the XML file to see what it is parsing and trying to upload? btw, immediately afterwards, there is this: which looks bad. (fwiw, there is one other "Traceback" case in the log file, but it looks like that one possibly is expected -- it would be nice if there was some output to make that very explicit. Or, to suppress this "scary" output in this case.) |
|
One of the AzDO logs shows: but I don't know where that ends up. |
If we're talking about the most recent AzDO link you pasted, it's on the artifacts tab: ... if I'm mistaken and you're referencing a different pipeline execution, please share its link. |
|
@MattGal Thanks -- the AzDO UI changed and I couldn't find the artifacts. Now I found them, so I see this. Obviously, this folder does not include any The log file I referenced was: (1) download all log files (looks like there isn't a deep link into these), (2) "CoreCLR Pri1 Test Run Linux_musl arm64 checked" folder, (3) "19_Publish Logs.txt" file: So that's presumably how the log files from the machine invoking tests in Helix get updated. From the Helix console log (above), Who is supposed to copy It looks like the answer is |
|
Ok, I ran manually and got a testRun.xml file (not sure who chooses this name or why it's not testResults.xml). For a particular failure case, I see: So while the "correct" text is there, the "escaped" failure message text that is used by the xunit.py parser (in run.py) is apparently coming from One workaround would be to use the The other avenue is to figure out why xunit (?) is doing this newline escaping. Comments? |
|
@ViktorHofer Looks like you've worked on the xunit console runner. Do we still use the one in the arcade repo? Assuming we use the xunit from https://github.com/xunit/xunit (or some close cousin), the answer seems to be in src\xunit.runner.utility\Visitors\XmlTestExecutionVisitor.cs: Interestingly, in that same file, when creating the "output" item (also a so, no Maybe none of the CDATA elements should use I would suggest simply un-escaping this in the xunit.py script that is parsing these, but that has the potential to corrupt legal pre-existing cases of |
|
We could unescape them, but I don't like it messing up paths and other places \r or \n shows up. A CDATA section shouldn't have any escaping in it at all. Newlines in xml don't even need to be escaped. Even attribute values can have literal newlines in them. I don't think the xunit console runner is in active development so a change there would be hard to get published. If you can switch to using |
|
We use the Microsoft.DotNet.XUnitConsoleRunner which lives in Arcade for netcoreapp runs. This is only the runner, so sources from xunit.runner.utility are not fixable in Arcade but in xunit/xunit. We plan to switch to |
|
+1 for wanting a fix for this. Makes looking at test output for new contributors a terrible experience. If |
|
The |
ViktorHofer
removed
the
untriaged
When a test fails, and you view the Error Message in Azure DevOps, you get something like this:
This is unreadable. We need proper line breaking in the view instead of the embedded
\ncharacters.@dotnet/runtime-infrastructure
The text was updated successfully, but these errors were encountered: