Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt #72977

VincentBu · 2022-07-28T01:20:53Z

Test affected:

CancelPendingRequest_DropsStalledConnectionAttempt - System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt

Failures 4/1-8/3 (incl. PRs):

57 failures (~8 per day), started on 7/27 only in official runs - amd64, arm and arm64

let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('', 'CancelPendingRequest_DropsStalledConnectionAttempt', true, 'RemoteExecutionException', true);

Original report

Run: runtime-libraries-coreclr outerloop 20220727.7

Failed test：

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.11.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-11-helix-arm32v7-20210304164347-5a7c380

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.10.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-arm32v7-20210304164340-6616c63

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.2204.Amd64.Open

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.


Stack trace

Child exception:
  System.AggregateException: One or more errors occurred. (A task was canceled.) ( Received an unexpected EOF or 0 bytes from the transport stream.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 632
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 551
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.<>c__DisplayClass2_0.<<CancelPendingRequest_DropsStalledConnectionAttempt_Impl>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 66
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 38
--- End of stack trace from previous location ---
   at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 234
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt_Impl(String versionString, String firstConnectionDelayMsString, String requestTimeoutMsString) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 52
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 65
 ---> (Inner Exception #1) System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 363
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 289
   at System.Net.Test.Common.Http2LoopbackConnection.CreateAsync(SocketWrapper socket, Stream stream, Http2Options httpOptions, TimeSpan timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 72
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Nullable`1 timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 99
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionGetSettingsAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 130
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 119
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Func`2 funcAsync) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 159
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.<>c.<<CancelPendingRequest_DropsStalledConnectionAttempt_Impl>b__2_1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 71
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120<---


Child process:
  System.Net.Http.Functional.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel System.Threading.Tasks.Task CancelPendingRequest_DropsStalledConnectionAttempt_Impl(System.String, System.String, System.String)

Child arguments:
  2.0, 10000, 1000

The text was updated successfully, but these errors were encountered:

ghost · 2022-07-28T01:21:02Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Run: runtime-libraries-coreclr outerloop 20220727.7

Failed test：

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.11.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-11-helix-arm32v7-20210304164347-5a7c380

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.10.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-10-helix-arm32v7-20210304164340-6616c63

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.2204.Amd64.Open

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.


Stack trace

Child exception:
  System.AggregateException: One or more errors occurred. (A task was canceled.) ( Received an unexpected EOF or 0 bytes from the transport stream.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 632
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 551
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.<>c__DisplayClass2_0.<<CancelPendingRequest_DropsStalledConnectionAttempt_Impl>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 66
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 38
--- End of stack trace from previous location ---
   at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 234
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt_Impl(String versionString, String firstConnectionDelayMsString, String requestTimeoutMsString) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 52
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 65
 ---> (Inner Exception #1) System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 363
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 289
   at System.Net.Test.Common.Http2LoopbackConnection.CreateAsync(SocketWrapper socket, Stream stream, Http2Options httpOptions, TimeSpan timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 72
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Nullable`1 timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 99
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionGetSettingsAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 130
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 119
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Func`2 funcAsync) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 159
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.<>c.<<CancelPendingRequest_DropsStalledConnectionAttempt_Impl>b__2_1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 71
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120<---


Child process:
  System.Net.Http.Functional.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel System.Threading.Tasks.Task CancelPendingRequest_DropsStalledConnectionAttempt_Impl(System.String, System.String, System.String)

Child arguments:
  2.0, 10000, 1000

Author:	VincentBu
Assignees:	-
Labels:	`area-System.Net.Http`
Milestone:	-

karelz · 2022-08-03T11:27:12Z

Triage: Looks like a recent regression -- happening 8x per day. We should disable the test ASAP - @rzikm can you please do it? Thanks!

antonfirsov · 2022-08-03T20:20:03Z

This is test from #71785. It's only happening on Ubuntu and only for HTTP 2.0.

wfurt · 2022-08-03T22:01:33Z

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.11.Arm32.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-11-helix-arm32v7-20210304164347-5a7c380

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

while Ubuntu is base image, the test is actually running on Debian.

antonfirsov · 2022-08-05T01:35:18Z

@shimingsg that is a different, unrelated test: #73343

karelz · 2022-08-07T14:34:57Z

Test disabled in PR #73300

karelz · 2022-08-12T17:45:46Z

Test is disabled, not blocking CI anymore - removing 'blocking-outerloop' label.

dotnet-issue-labeler bot added the area-System.Net.Http label Jul 28, 2022

ghost added the untriaged New issue has not been triaged by the area owner label Jul 28, 2022

VincentBu added os-linux Linux OS (any supported distro) arch-x64 blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs labels Jul 28, 2022

runfoapp bot mentioned this issue Jul 28, 2022

Infrastructure - Status/Health #702

Closed

karelz assigned rzikm Aug 2, 2022

karelz removed the arch-x64 label Aug 3, 2022

karelz removed the untriaged New issue has not been triaged by the area owner label Aug 3, 2022

karelz added this to the 7.0.0 milestone Aug 3, 2022

rzikm mentioned this issue Aug 3, 2022

Disable System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt on Linux #73300

Merged

rzikm added the disabled-test The test is disabled in source code against the issue label Aug 4, 2022

rzikm removed their assignment Aug 4, 2022

This comment was marked as off-topic.

Sign in to view

karelz removed the blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs label Aug 12, 2022

karelz assigned MihaZupan Aug 23, 2022

MihaZupan mentioned this issue Aug 23, 2022

Rewrite PendingConnectionTimeout tests #74427

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 23, 2022

MihaZupan closed this as completed in #74427 Aug 23, 2022

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Aug 23, 2022

karelz modified the milestones: 7.0.0, 8.0.0 Aug 26, 2022

ghost locked as resolved and limited conversation to collaborators Sep 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt #72977

Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt #72977

VincentBu commented Jul 28, 2022 •

edited by karelz

Loading

ghost commented Jul 28, 2022

karelz commented Aug 3, 2022

antonfirsov commented Aug 3, 2022

wfurt commented Aug 3, 2022

This comment was marked as off-topic.

antonfirsov commented Aug 5, 2022

karelz commented Aug 7, 2022

karelz commented Aug 12, 2022 •

edited

Loading

Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt #72977

Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt #72977

Comments

VincentBu commented Jul 28, 2022 • edited by karelz Loading

Original report

ghost commented Jul 28, 2022

karelz commented Aug 3, 2022

antonfirsov commented Aug 3, 2022

wfurt commented Aug 3, 2022

This comment was marked as off-topic.

antonfirsov commented Aug 5, 2022

karelz commented Aug 7, 2022

karelz commented Aug 12, 2022 • edited Loading

VincentBu commented Jul 28, 2022 •

edited by karelz

Loading

karelz commented Aug 12, 2022 •

edited

Loading