Add Wasm testing in Chromium, not just command-line `v8` #40786

directhex · 2020-08-13T16:56:26Z

This should give us a better idea of any cases where browser and JS engine behaviour diverge.

…indows

…rowsening

…ASM on Windows" This reverts commit 95706be.

…rowsening

Dotnet-GitSync-Bot · 2020-08-13T16:56:29Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2020-08-13T16:57:57Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

directhex · 2020-08-13T18:05:48Z

Failures are because https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/9677 hasn't landed in prod Helix instance yet.

premun · 2020-08-17T11:44:46Z

eng/pipelines/runtime.yml

@@ -334,6 +334,11 @@ jobs:
      extraStepsParameters:
        creator: dotnet-bot
        testRunNamePrefixSuffix: Mono_$(_BuildConfig)
+        scenarios:
+        # Only one scenario per build for jobs which install xharness CLI
+        # until we fix https://github.com/dotnet/arcade/issues/5919


dotnet/arcade#5919 should be fixed now, but you might need to update the Helix SDK since Arcade wasn't released yet

I tried bumping Arcade as part of this for testing and EVERYTHING EXPLODED so I'll wait for a bump to hit master I think

@directhex bump Helix SDK, not arcade. Check the iOS draft PR and cherry pick that

…rowsening

This reverts commit 10243dc.

…rowsening

directhex · 2020-08-21T15:53:38Z

…rowsening

akoeplinger · 2020-08-31T14:26:07Z

eng/testing/tests.mobile.targets

@@ -14,7 +14,7 @@

  <PropertyGroup Condition="'$(TargetOS)' == 'Browser'">
    <!-- We need to set this in order to get extensibility on xunit category traits and other arguments we pass down to xunit via MSBuild properties -->
-    <RunScriptCommand>$HARNESS_RUNNER wasm test --engine=$(JSEngine) $(JSEngineArgs) --js-file=runtime.js -v --output-directory=$XHARNESS_OUT -- --run WasmTestRunner.dll $(AssemblyName).dll</RunScriptCommand>
+    <RunScriptCommand>$HARNESS_RUNNER wasm test$TESTINBROWSER --app=. --engine=$(JSEngine) $(JSEngineArgs) --js-file=runtime.js -v --output-directory=$XHARNESS_OUT -- --run WasmTestRunner.dll $(AssemblyName).dll</RunScriptCommand>


nit: I'd prefer substituting the whole command rather than appending only the -browser portion.

E.g.

Suggested change

<RunScriptCommand>$HARNESS_RUNNER wasm test$TESTINBROWSER --app=. --engine=$(JSEngine) $(JSEngineArgs) --js-file=runtime.js -v --output-directory=$XHARNESS_OUT -- --run WasmTestRunner.dll $(AssemblyName).dll</RunScriptCommand>

<RunScriptCommand>$HARNESS_RUNNER wasm $XHARNESS_COMMAND --app=. --engine=$(JSEngine) $(JSEngineArgs) --js-file=runtime.js -v --output-directory=$XHARNESS_OUT -- --run WasmTestRunner.dll $(AssemblyName).dll</RunScriptCommand>

and then setting XHARNESS_COMMAND to test or test-browser depending on Scenario.

…rowsening

steveisok · 2020-09-16T16:10:52Z

Definitely looks like we're still blocked by #41160 and #41269.

.. on X509 certificates. Tests discovery fails on `System.Net.Http.Functional.Tests.HttpClientEKUTest..cctor()` which ends up throwing PNSE for `Certificates cctor threw System.PlatformNotSupportedException: System.Security.Cryptography.X509Certificates is not supported on this platform.` Because it's a static ctor, `SkipOnMono` will not be helpful to skip this one. So, we use `#if TARGETS_BROWSER` instead.

radical · 2020-09-25T19:04:04Z

Should I open a separate PR for fixing System.Net.Http/tests/FunctionalTests? Or add it here? https://gist.github.com/radical/f7f1a99ac9f4bd33554056401fdfeb4f

steveisok · 2020-09-25T19:40:10Z

I'd probably just add it here.

… in the browser vs v8

…untime into add_browser_2_the_browsening

- This showed up when running System.Net.Http's FunctionalTests, as creating a listening socket fails there with `System.PlatformNotSupportedException : System.Net.Sockets is not supported on this platform.` - Also, surface the exception so it can correctly fail tests.

System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClient.HandlerTest.GetAsync_InvalidUrl_ExpectedExceptionThrown

`SocketsHttpHandler_HttpClientHandlerTest`

radical · 2020-09-26T07:20:53Z

System.Net.Http's FunctionalTests can now be enabled by changing SkipOnMono in src/libraries/System.Net.Http/tests/FunctionalTests/AssemblyInfo.cs.

And dotnet/xharness#313 - has some nice additions.

lewing · 2020-09-27T00:01:14Z

src/mono/wasm/runtime-test.js


 if (is_browser) {
 	// We expect to be run by tests/runtime/run.js which passes in the arguments using http parameters
+	window.real_print = console.log;


I want this in asap but after it lands I think we should consider sending all this as json over the websocket and expand it on the xharness side, we should be able to clean up the console output considerably.

pseudo code like

JSON.stringify ({ 'method': method, args: {...args}})

where method is { 'print', 'console.debug', ... }

and after that it might make sense to add a custom log provider to preserve even more structure.

lewing · 2020-09-27T00:03:22Z

src/mono/wasm/runtime-test.js

+
+	consoleWebSocket = new WebSocket(consoleUrl);
+	consoleWebSocket.onopen = function(event) {
+		consoleWebSocket.send("browser: Console websocket connected.");


This might be a better place to force everything through print? I sometimes use runtime-tests.js outside of xharness

Yeah, that makes sense. And I was thinking maybe the WebSocket thing should be optional? I found it useful to run the tests without the harness, directly in a browser.

I think it is worth a follow-up once we have everything working in CI.

Update from channel - .NET Eng - Latest Updating 'Microsoft.DotNet.XHarness.CLI': '1.0.0-prerelease.20474.1' => '1.0.0-prerelease.20476.2' (from build '20200926.2' of 'https://github.com/dotnet/xharness') Updating 'Microsoft.DotNet.XHarness.TestRunners.Xunit': '1.0.0-prerelease.20474.1' => '1.0.0-prerelease.20476.2' (from build '20200926.2' of 'https://github.com/dotnet/xharness')

radical · 2020-09-27T06:32:12Z

1 failing test remaining:

      [FAIL] System.Xml.Tests.AsyncReaderLateInitTests.ReadAsyncAfterInitializationWithUriThrows

      System.InvalidOperationException : An asynchronous operation is already in progress.     
         at System.Xml.XmlAsyncCheckReader.CheckAsync()
         at System.Xml.XmlAsyncCheckReader.Dispose(Boolean disposing)
         at System.Xml.XmlReader.Dispose()
         at System.Xml.Tests.AsyncReaderLateInitTests.ReadAsyncAfterInitializationWithUriThrows()
         at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

steveisok · 2020-09-28T21:01:02Z

@radical I'm ok with skipping the last failing test.

@akoeplinger Can you give this a final review to move things forward?

…now too

lewing · 2020-09-28T22:16:13Z

@steveisok can we review System.Xml.Tests.AsyncReaderLateInitTests the for the actual browser case after this.

akoeplinger · 2020-09-29T13:45:17Z

src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.cs

@@ -941,6 +943,7 @@ public sealed class SocketsHttpHandler_SchSendAuxRecordHttpTest : SchSendAuxReco
        public SocketsHttpHandler_SchSendAuxRecordHttpTest(ITestOutputHelper output) : base(output) { }
    }

+    [SkipOnMono("Tests hang with chrome. To be investigated", TestPlatforms.Browser)]


do we have a tracking issue for this?

Should any of the SocketsHttpHandler tests be working?

No, SocketsHttpHandler should throw PNSE on wasm-browser

Ok, let's follow up on this.

lewing · 2020-09-29T17:52:45Z

Opened #42852 to discuss the loopback server problem

directhex added 9 commits August 6, 2020 14:53

Initial target to put Chromium into correlation payload for WASM on W…

95706be

…indows

Merge remote-tracking branch 'origin/master' into add_browser_2_the_b…

778a755

…rowsening

Revert "Initial target to put Chromium into correlation payload for W…

ce806e5

…ASM on Windows" This reverts commit 95706be.

Save intermediate state

673fb32

Bump to xharness version with test wasm-browser support

ba3451d

Fix running wasm tests in a browser

0fe0f75

Bump xharness again, for selenium fix

237fe1b

Specify browser scenario for wasm testing

e666013

Merge remote-tracking branch 'origin/master' into add_browser_2_the_b…

9f3d73c

…rowsening

directhex added the area-Infrastructure-mono label Aug 13, 2020

premun reviewed Aug 17, 2020

View reviewed changes

directhex added 7 commits August 17, 2020 09:01

Merge remote-tracking branch 'origin/master' into add_browser_2_the_b…

e08cffe

…rowsening

Bump Arcade to fix Helix bug

10243dc

Revert "Bump Arcade to fix Helix bug"

aec3f25

This reverts commit 10243dc.

Try bumping Helix SDK again

ffc5c9d

Merge remote-tracking branch 'origin/master' into add_browser_2_the_b…

ef74cd6

…rowsening

Bump xharness, for supportFiles in Chromium fix

f2914bf

Merge remote-tracking branch 'origin/master' into add_browser_2_the_b…

06ea87b

…rowsening

directhex added 3 commits August 24, 2020 09:07

Bump xharness, for timeout fix

571fa98

Merge remote-tracking branch 'origin/master' into add_browser_2_the_b…

a5175b1

…rowsening

Merge remote-tracking branch 'origin/master' into add_browser_2_the_b…

7cf7cb8

…rowsening

akoeplinger reviewed Aug 31, 2020

View reviewed changes

MaximLipnin mentioned this pull request Sep 7, 2020

Disable System.Xml.Tests.AsyncReaderLateInitTests.InitializationWithUriOnNonAsyncReaderTrows test on Browser WASM due to PNSE #41936

Merged

directhex added 2 commits September 15, 2020 13:01

Merge remote-tracking branch 'origin/master' into add_browser_2_the_b…

06540e1

…rowsening

Wholesale swap-out of test command for wasm, not null-or-suffix fiddling

e1be91b

Steve Pfister and others added 6 commits September 25, 2020 18:05

Added environment variable to help platform detect when we're running…

dea619b

… in the browser vs v8

Merge branch 'add_browser_2_the_browsening' of github.com:directhex/r…

1a52cad

…untime into add_browser_2_the_browsening

[wasm][tests] Don't include HttpClientEKUTest.cs for browser tests

becbfe7

[wasm][tests] Disable a hanging System.Net.Http test

d8100a6

System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClient.HandlerTest.GetAsync_InvalidUrl_ExpectedExceptionThrown

[wasm][tests] Disable System.Net.Http tests hanging with the browser

4521654

`SocketsHttpHandler_HttpClientHandlerTest`

lewing reviewed Sep 27, 2020

View reviewed changes

This was referenced Sep 28, 2020

Agents canceling during dotnet builds jaredpar/runfo#47

Closed

Azure Agent timeout dotnet/roslyn#48030

Closed

lewing mentioned this pull request Sep 28, 2020

[browser][http] Set HttpResponseMessage.RequestMessage #42822

Merged

Skip ReadAsyncAfterInitializationWithUriThrows without threading for …

1317668

…now too

steveisok self-requested a review September 28, 2020 22:09

steveisok approved these changes Sep 28, 2020

View reviewed changes

akoeplinger approved these changes Sep 28, 2020

View reviewed changes

akoeplinger added 2 commits September 29, 2020 03:33

Merge branch 'master' into add_browser_2_the_browsening

90b7097

Rethrow exception in LoopbackServer

145722a

akoeplinger reviewed Sep 29, 2020

View reviewed changes

akoeplinger merged commit b25c7c4 into dotnet:master Sep 29, 2020

This was referenced Sep 30, 2020

System.Threading.Tasks.Tests timed out on net5.0-Linux-Debug-arm64-Mono_release #42024

Closed

Builds legs getting abandoned due to agent notification issues #35223

Closed

ghost locked as resolved and limited conversation to collaborators Dec 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Wasm testing in Chromium, not just command-line `v8` #40786

Add Wasm testing in Chromium, not just command-line `v8` #40786

directhex commented Aug 13, 2020

Dotnet-GitSync-Bot commented Aug 13, 2020

ghost commented Aug 13, 2020

directhex commented Aug 13, 2020

premun Aug 17, 2020

directhex Aug 17, 2020

premun Aug 17, 2020

directhex commented Aug 21, 2020 •

edited

Loading

akoeplinger Aug 31, 2020

steveisok commented Sep 16, 2020

radical commented Sep 25, 2020

steveisok commented Sep 25, 2020

radical commented Sep 26, 2020

lewing Sep 27, 2020 •

edited

Loading

lewing Sep 27, 2020

radical Sep 27, 2020

lewing Sep 27, 2020

radical commented Sep 27, 2020

steveisok commented Sep 28, 2020

lewing commented Sep 28, 2020 •

edited

Loading

akoeplinger Sep 29, 2020

lewing Sep 29, 2020

marek-safar Sep 29, 2020

akoeplinger Sep 29, 2020

lewing commented Sep 29, 2020

	<RunScriptCommand>$HARNESS_RUNNER wasm test$TESTINBROWSER --app=. --engine=$(JSEngine) $(JSEngineArgs) --js-file=runtime.js -v --output-directory=$XHARNESS_OUT -- --run WasmTestRunner.dll $(AssemblyName).dll</RunScriptCommand>
	<RunScriptCommand>$HARNESS_RUNNER wasm $XHARNESS_COMMAND --app=. --engine=$(JSEngine) $(JSEngineArgs) --js-file=runtime.js -v --output-directory=$XHARNESS_OUT -- --run WasmTestRunner.dll $(AssemblyName).dll</RunScriptCommand>

Add Wasm testing in Chromium, not just command-line v8 #40786

Add Wasm testing in Chromium, not just command-line v8 #40786

Conversation

directhex commented Aug 13, 2020

Dotnet-GitSync-Bot commented Aug 13, 2020

ghost commented Aug 13, 2020

directhex commented Aug 13, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

directhex commented Aug 21, 2020 • edited Loading

Choose a reason for hiding this comment

steveisok commented Sep 16, 2020

radical commented Sep 25, 2020

steveisok commented Sep 25, 2020

radical commented Sep 26, 2020

lewing Sep 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

radical commented Sep 27, 2020

steveisok commented Sep 28, 2020

lewing commented Sep 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewing commented Sep 29, 2020

Add Wasm testing in Chromium, not just command-line `v8` #40786

Add Wasm testing in Chromium, not just command-line `v8` #40786

directhex commented Aug 21, 2020 •

edited

Loading

lewing Sep 27, 2020 •

edited

Loading

lewing commented Sep 28, 2020 •

edited

Loading