System.Net.Security test stabilization

[CIPop]

• Adding generic parallel test execution script.
• Address several System.Net.Security test issues found using the script.

The following tests have been used:

• 100 parallel test processes on a 12CPU/16GB machine for 1hr
• 20 parallel test processes on a 8CPU/3GB machine (leaving this to run overnight)
• 20 parallel test processes on a 2CPU/2GB machine (leaving this to run overnight)

Most of below bugs were potentially fixed by #7800 at least in part. Other test-code fixes to issues discovered during stress tests have been added to this PR.

Fixes #4467 #5283 #5284 #5991.

@davidsh @stephentoub @ericeil PTAL
/cc @himadrisarkar

[CIPop]

Unix tests failed as I forgot to change the test unix/project.json file.

[CIPop]

The OSX package restore failed, potentially caused by #7902:

16:46:06 EXEC : error : Could not find file '/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx_debug_prtest/tempHome/.local/share/NuGet/v3-cache/0b99e17097bea891c4215989667b88d90c9f7540$_F_dotnet-core_api_v3_index.json/nupkg_System.Runtime.InteropServices.4.1.0-rc3-24018-00.dat'. [/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx_debug_prtest/build.proj]
16:46:06 
/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx_debug_prtest/build.proj(80,5): error MSB3073: The command ""/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx_debug_prtest/Tools/dotnetcli/dotnet" restore --packages "/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx_debug_prtest/packages" --source https://dotnet.myget.org/F/dotnet-core/api/v3/index.json --source https://www.nuget.org/api/v2/ --infer-runtimes "/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx_debug_prtest/src" "/Users/dotnet-bot/j/workspace/dotnet_corefx/master/osx_debug_prtest/pkg"" exited with code -1.
16:46:06 
16:46:06     0 Warning(s)
16:46:06     2 Error(s)

[CIPop]

@dotnet-bot Test Innerloop OSX Debug Build and Test

[davidsh]

LGTM

[stephentoub]

Nit: since this appears to be the total number of tries rather than retries, maybe remainingTries

[CIPop]

Fixing.

[CIPop]

Overnight test results. One test failed:

System.Net.Security.Tests.CertificateValidationRemoteServer.CertificateValidationRemoteServer_EndToEnd_Ok [FAIL]
      System.Net.Sockets.SocketException : A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
      Stack Trace:
            at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
            at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
            at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
         C:\corefx\src\System.Net.Security\tests\FunctionalTests\CertificateValidationRemoteServer.cs(21,0): at System.Net.Security.Tests.CertificateValidationRemoteServer.<CertificateValidationRemoteServer_EndToEnd_Ok>d__0.MoveNext()
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
         --- End of stack trace from previous location where exception was thrown ---
            at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
            at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
Finished:    System.Net.Security.Tests

=== TEST EXECUTION SUMMARY ===
   System.Net.Security.Tests  Total: 102, Errors: 0, Failed: 1, Skipped: 0, Time: 24.210s
Test execution failed!

• 20 parallel test processes on a 8CPU/3GB machine (leaving this to run overnight): 2/20 test processes failed with above error.
• 20 parallel test processes on a 2CPU/2GB machine (leaving this to run overnight): 15/20 test processes failed with above error.

/cc @davidsh This particular test is the only one having a dependency on Azure. What we might have seen was either downtime or automatic DoS countermeasure implemented on the remote server.

Tracking with #3697.

[CIPop]

Same failures on both OSX boxes (#7902).

[CIPop]

No failures on both the VMs so far with the current configuration.
Changing the positive test timeout back to 15s now.

[CIPop]

Changing the positive test timeout back to 15s now.

@stephentoub No-go: tests failed within the first minute of 100 parallel executions on my 12 CPU machine. It appears that the tests do run longer than 10s for the first time if a lot of processes are created. After a while they enter a steady-state where tests run fast (7-8s).

I've changed the timeout to 1 minute to avoid the timeouts in CI.

[CIPop]

Overnight tests failed the same way: System.Net.Security.Tests.CertificateValidationRemoteServer.CertificateValidationRemoteServer_EndToEnd_Ok [FAIL].
@stephentoub please let me know if there are any other changes that I should make before merging.

[CIPop]

Ubuntu test failed because of #7234

[CIPop]

Test Innerloop Ubuntu14.04 Debug Build and Test please