Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HTTP/3] Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Cancellation_Test_Http3_Mock.ConnectTimeout_TimesOutSSLAuth_Throws #54806

Closed
VincentBu opened this issue Jun 28, 2021 · 13 comments · Fixed by #54825 or #54843
Closed

[HTTP/3] Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Cancellation_Test_Http3_Mock.ConnectTimeout_TimesOutSSLAuth_Throws #54806

VincentBu opened this issue Jun 28, 2021 · 13 comments · Fixed by #54825 or #54843
Assignees
@geoffkizer
Labels
area-System.Net.Http test-run-core Test failures in .NET Core test runs
Milestone
6.0.0

Comments

@VincentBu
Copy link
Contributor

Run: runtime-libraries-coreclr outerloop 20210627.4

Failed test:

net6.0-Linux-Release-arm-CoreCLR_release-(Debian.9.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-9-helix-arm32v7-bfcd90a-20200121150037

- System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Cancellation_Test_Http3_Mock.ConnectTimeout_TimesOutSSLAuth_Throws

Error message:

Assert.Throws() Failure
Expected: typeof(System.OperationCanceledException)
Actual:   typeof(System.Net.Http.HttpRequestException): Connection refused (127.0.0.1:1)
---- System.Net.Http.HttpRequestException : Connection refused (127.0.0.1:1)
-------- System.Net.Sockets.SocketException : Connection refused


Stack trace
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1355
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1263
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 525
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 822
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 884
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 901
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 920
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
----- Inner Stack Trace -----
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1355
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1263
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 525
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 822
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 884
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 901
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 92
@ghost
Copy link

ghost commented Jun 28, 2021

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-libraries-coreclr outerloop 20210627.4

Failed test:

net6.0-Linux-Release-arm-CoreCLR_release-(Debian.9.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-9-helix-arm32v7-bfcd90a-20200121150037

- System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Cancellation_Test_Http3_Mock.ConnectTimeout_TimesOutSSLAuth_Throws

Error message:

Assert.Throws() Failure
Expected: typeof(System.OperationCanceledException)
Actual:   typeof(System.Net.Http.HttpRequestException): Connection refused (127.0.0.1:1)
---- System.Net.Http.HttpRequestException : Connection refused (127.0.0.1:1)
-------- System.Net.Sockets.SocketException : Connection refused


Stack trace
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1355
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1263
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 525
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 822
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 884
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 901
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 920
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs:line 30
----- Inner Stack Trace -----
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1355
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 1263
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 525
   at System.Net.Http.HttpConnectionPool.TrySendUsingHttp2Async(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 822
   at System.Net.Http.HttpConnectionPool.DetermineVersionAndSendAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 884
   at System.Net.Http.HttpConnectionPool.SendAndProcessAltSvcAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 901
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs:line 92
Author: VincentBu
Assignees: -
Labels:

arch-arm32, arch-arm64, arch-x64, arch-x86, area-System.Net.Http, os-linux, os-mac-os-x, os-windows
Milestone: -

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jun 28, 2021
@wfurt
Copy link
Member

wfurt commented Jun 28, 2021

The port is really strange 127.0.0.1:1 => 1 is reserved port.

@karelz karelz changed the title Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Cancellation_Test_Http3_Mock.ConnectTimeout_TimesOutSSLAuth_Throws [HTTP/3] Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandler_Cancellation_Test_Http3_Mock.ConnectTimeout_TimesOutSSLAuth_Throws Jun 28, 2021
@karelz
Copy link
Member

karelz commented Jun 28, 2021

Failures 4/28-6/27 (incl. PRs):

Date Run Details
6/26 PR #54755 13 failures
6/26 Official run 39 failures
6/27 Official run 39 failures

We need to disable the tests ASAP.

@karelz karelz added the test-run-core Test failures in .NET Core test runs label Jun 28, 2021
@karelz
Copy link
Member

karelz commented Jun 28, 2021

@ManickaP @wfurt @geoffkizer any thoughts how to disable the test just for HTTP/3?

@karelz karelz added this to the 6.0.0 milestone Jun 28, 2021
@wfurt
Copy link
Member

wfurt commented Jun 28, 2021

is everything on ARM @karelz ?

@karelz
Copy link
Member

karelz commented Jun 28, 2021

Nope, arm, amd64, all OS versions - Windows, Linux, OSX

@ManickaP
Copy link
Member

The port "1" in the exception looks suspicious. Are we really using TCPMUX? Or is this some magic in our test environment and it should "just work" tm?

@ManickaP
Copy link
Member

ManickaP commented Jun 28, 2021 •
edited
Loading

Even more, this is SocketsHttpHandler_HttpClientHandler_Cancellation_Test_Http3_Mock and mocks are using in memory buffers. There shouldn't even be any networking involved. This is suspicious.

@wfurt
Copy link
Member

wfurt commented Jun 28, 2021

typically the server would bind on anonymous port (e.g. 0) and then OS would assign some real unused port. Then we would tell client where to connect to. I'm not sure what exactly we do for mock.

@ManickaP
Copy link
Member

I'm not sure what exactly we do for mock.

Nothing, we don't do any networking, we use in-memory buffers to exchange data between client and server.

@ManickaP
Copy link
Member

This might be the problem:

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.cs

Line 35 in cef40a1

using (var handler = new SocketsHttpHandler())

It basically walks around the whole handler set up we do for testing clients:

runtime/src/libraries/System.Net.Http/tests/FunctionalTests/HttpClientHandlerTestBase.SocketsHttpHandler.cs

Lines 38 to 57 in cef40a1

protected static HttpClientHandler CreateHttpClientHandler(Version useVersion = null, QuicImplementationProvider quicImplementationProvider = null, bool allowAllHttp2Certificates = true)
{
useVersion ??= HttpVersion.Version11;
HttpClientHandler handler = (PlatformDetection.SupportsAlpn && useVersion != HttpVersion.Version30) ? new HttpClientHandler() : new VersionHttpClientHandler(useVersion);
if (useVersion >= HttpVersion.Version20 && allowAllHttp2Certificates)
{
handler.ServerCertificateCustomValidationCallback = TestHelper.AllowAllCertificates;
}
if (quicImplementationProvider != null)
{
SocketsHttpHandler socketsHttpHandler = (SocketsHttpHandler)GetUnderlyingSocketsHttpHandler(handler);
SetQuicImplementationProvider(socketsHttpHandler, quicImplementationProvider);
socketsHttpHandler.SslOptions.RemoteCertificateValidationCallback = (sender, certificate, chain, errors) => true;
}
return handler;
}

This should be quick, I'll put up a PR. If it helps, great, if not, I'll disable the test.

@ManickaP ManickaP mentioned this issue Jun 28, 2021
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Jun 28, 2021
@geoffkizer
Copy link
Contributor

I'm not sure what exactly we do for mock.

We have to create a fake IPEndPoint for Mock. That's where the :1 is coming from.

The problem is, we seem to have a mismatch in this test where we are using Mock on the loopback server but MsQuic on the client. So we end up actually trying to connect to the fake endpoint.

The reason for the mismatch is what @ManickaP pointed out above -- we are constructing the SocketsHttpHandler directly instead of using CreateHttpClientHandler. As a result it doesn't get configured to use Mock.

Interestingly, this test seems to run fine on Windows. I guess the difference here is that Linux is actively denying an attempt to connect to 127.0.0.1:1 whereas Windows is not.

@geoffkizer
Copy link
Contributor

This is a weird test because it's intentionally not completing the connection -- specifically, it's written so that the connection should be accepted but the SSL establishment shouldn't happen.

This doesn't really make sense for HTTP3. I think we should just disable this test for HTTP3, like we do the one just below.

@geoffkizer geoffkizer mentioned this issue Jun 28, 2021
Merged
@karelz karelz removed the untriaged New issue has not been triaged by the area owner label Jun 29, 2021
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Jun 30, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Jul 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@geoffkizer geoffkizer

Labels
area-System.Net.Http test-run-core Test failures in .NET Core test runs
Projects
None yet
Milestone
6.0.0
5 participants
@karelz @ManickaP @geoffkizer @wfurt @VincentBu