Mono crash in System.Net.Requests tests on linux-arm64 #74667

steveisok · 2022-08-26T17:52:30Z

It appears that there is some trouble trying to reap child processes and that leads to a crash. Could it be possible we are timing out waiting for a request to finish?

Example log

Process terminated.
Error while reaping child. errno = 10

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5589c036fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7f92ba5ff0 (LWP 25) "dotnet" 0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5589c036fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0x7f91bff1c0 (LWP 26) "SGen worker" 0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f926008c8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7f8fe0f1c0 (LWP 27) "dotnet" 0x0000007f92841ef8 in __GI___poll (fds=0x7f88003af0, nfds=547913473925, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0x7f8fc0e1c0 (LWP 28) "Finalizer" 0x0000007f92b69a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f925f19c0 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7f8ccd51c0 (LWP 29) "dotnet" 0x0000007f92b6bd5c in __waitpid (pid=<optimized out>, stat_loc=0x7f8ccd22b0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30

Lot of other threads are waiting similar to

Thread 14 (Thread 0x7f76bf91c0 (LWP 40)):
#0  0x0000007f92b675bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f00000000, expected=0, futex_word=0x7f5c00547c) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0x7f76bf3fc8, mutex=0x7f5c005420, cond=0x7f5c005450) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0x7f5c005450, mutex=0x7f5c005420, abstime=0x7f76bf3fc8) at pthread_cond_wait.c:667
#3  0x0000007f8f768490 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f5c005420, timeoutMilliseconds=30000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000007f8c1f958c in ?? ()
#5  0x0000007f8cbc52c8 in ?? ()
#6  0x0000007f8fe7b1a0 in ?? ()
#7  0xf90047bda907f3bb in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

{ "ErrorMessage":"Error while reaping child. errno = 10" }

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

The text was updated successfully, but these errors were encountered:

ghost · 2022-08-26T17:52:39Z

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

It appears that there is some trouble trying to reap child processes and that leads to a crash. Could it be possible we are timing out waiting for a request to finish?

Example log

Process terminated.
Error while reaping child. errno = 10

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5589c036fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7f92ba5ff0 (LWP 25) "dotnet" 0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5589c036fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0x7f91bff1c0 (LWP 26) "SGen worker" 0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f926008c8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7f8fe0f1c0 (LWP 27) "dotnet" 0x0000007f92841ef8 in __GI___poll (fds=0x7f88003af0, nfds=547913473925, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0x7f8fc0e1c0 (LWP 28) "Finalizer" 0x0000007f92b69a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f925f19c0 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7f8ccd51c0 (LWP 29) "dotnet" 0x0000007f92b6bd5c in __waitpid (pid=<optimized out>, stat_loc=0x7f8ccd22b0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30

Lot of other threads are waiting similar to

Thread 14 (Thread 0x7f76bf91c0 (LWP 40)):
#0  0x0000007f92b675bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f00000000, expected=0, futex_word=0x7f5c00547c) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0x7f76bf3fc8, mutex=0x7f5c005420, cond=0x7f5c005450) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0x7f5c005450, mutex=0x7f5c005420, abstime=0x7f76bf3fc8) at pthread_cond_wait.c:667
#3  0x0000007f8f768490 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f5c005420, timeoutMilliseconds=30000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000007f8c1f958c in ?? ()
#5  0x0000007f8cbc52c8 in ?? ()
#6  0x0000007f8fe7b1a0 in ?? ()
#7  0xf90047bda907f3bb in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Author:	steveisok
Assignees:	-
Labels:	`area-System.Threading`
Milestone:	-

The suite crashes on mono linux-arm64 and is being tracked by dotnet#74667

The suite crashes on mono linux-arm64 and is being tracked by #74667

ghost · 2022-08-30T20:11:31Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

It appears that there is some trouble trying to reap child processes and that leads to a crash. Could it be possible we are timing out waiting for a request to finish?

Example log

Process terminated.
Error while reaping child. errno = 10

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5589c036fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
88	../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
  Id   Target Id         Frame 
* 1    Thread 0x7f92ba5ff0 (LWP 25) "dotnet" 0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x5589c036fc) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  2    Thread 0x7f91bff1c0 (LWP 26) "SGen worker" 0x0000007f92b672a4 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f926008c8 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
  3    Thread 0x7f8fe0f1c0 (LWP 27) "dotnet" 0x0000007f92841ef8 in __GI___poll (fds=0x7f88003af0, nfds=547913473925, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
  4    Thread 0x7f8fc0e1c0 (LWP 28) "Finalizer" 0x0000007f92b69a40 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f925f19c0 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7f8ccd51c0 (LWP 29) "dotnet" 0x0000007f92b6bd5c in __waitpid (pid=<optimized out>, stat_loc=0x7f8ccd22b0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30

Lot of other threads are waiting similar to

Thread 14 (Thread 0x7f76bf91c0 (LWP 40)):
#0  0x0000007f92b675bc in futex_reltimed_wait_cancelable (private=<optimized out>, reltime=0x7f00000000, expected=0, futex_word=0x7f5c00547c) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#1  __pthread_cond_wait_common (abstime=0x7f76bf3fc8, mutex=0x7f5c005420, cond=0x7f5c005450) at pthread_cond_wait.c:533
#2  __pthread_cond_timedwait (cond=0x7f5c005450, mutex=0x7f5c005420, abstime=0x7f76bf3fc8) at pthread_cond_wait.c:667
#3  0x0000007f8f768490 in SystemNative_LowLevelMonitor_TimedWait (monitor=0x7f5c005420, timeoutMilliseconds=30000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195
#4  0x0000007f8c1f958c in ?? ()
#5  0x0000007f8cbc52c8 in ?? ()
#6  0x0000007f8fe7b1a0 in ?? ()
#7  0xf90047bda907f3bb in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

{ "ErrorMessage":"../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory." }

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Author:	steveisok
Assignees:	-
Labels:	`area-System.Net`, `disabled-test`, `os-linux`, `runtime-mono`
Milestone:	-

wfurt · 2022-08-30T20:22:49Z

this is not mono specific (even if mono failures seem prevalent) see #74795

steveisok · 2022-08-30T20:32:19Z

this is not mono specific (even if mono failures seem prevalent) see #74795

Thanks! I wondered but couldn't find any evidence.

wfurt · 2022-08-30T22:28:00Z

is there some way how to run the tests with Mono and dump the managed objects @steveisok? I would like to see if this is happening during some particular test....

steveisok · 2022-08-30T22:57:18Z

is there some way how to run the tests with Mono and dump the managed objects @steveisok? I would like to see if this is happening during some particular test....

I'm not sure. @akoeplinger would you happen to know?

akoeplinger · 2022-08-31T16:40:51Z

If you have the crash in lldb/gdb then https://www.mono-project.com/docs/debug+profile/debug/#debugging-with-gdb might help, maybe @BrzVlad knows a better way.

The only other quick suggestion I have to pinpoint the exact test is to write a custom xunit v2 assembly runner which logs when a test starts by subscribing to runner.OnTestStarting and setting the parallel parameter to false in the runner.Start() method: https://github.com/xunit/samples.xunit/blob/main/TestRunner/Program.cs

wfurt · 2022-08-31T16:48:52Z

We could disable parallelization as workaround for now. Any thoughts on that @karelz? Its seems better than disabling the tests....

karelz · 2022-09-01T10:56:42Z

I think this is showing in CI now quite a bit (I am behind on Test Monitor duty), in which case it is blocking CI and therefore any action to make it not a problem is good.
We can go after the root cause in parallel.

…4994) The test was using `RemoteExecutor.Invoke(Action<string, string, string, string> method, ...)` since there is no overload that takes `Func<string, string, string, string, Task>` (only one with three strings) and that means it's becoming an async void. The delegate that gets invoked will return the moment the method awaits something not yet completed, so now there's a race condition, where `RemoteExecutor.Invoke` thinks all work is done, but there's still likely work running and it'll start doing all its cleanup stuff like killing child processes. Fix by removing one string parameter so it picks the correct overload. I'll also open an arcade PR to add an overload with four string arguments. Fixes #74667

akoeplinger · 2022-09-05T12:56:34Z

Turns out this is still happening, will re-disable the tests and let's continue investigating in #74795 since it's the same issue

dotnet-issue-labeler bot added the area-System.Threading label Aug 26, 2022

ghost added the untriaged New issue has not been triaged by the area owner label Aug 26, 2022

steveisok added runtime-mono specific to the Mono runtime os-linux Linux OS (any supported distro) and removed untriaged New issue has not been triaged by the area owner labels Aug 26, 2022

steveisok pushed a commit to steveisok/runtime that referenced this issue Aug 26, 2022

Skip System.Net.Requests tests on mono linux-arm64

2a3ec8a

The suite crashes on mono linux-arm64 and is being tracked by dotnet#74667

This was referenced Aug 26, 2022

Skip System.Net.Requests tests on mono linux-arm64 #74668

Merged

[Android] Move linux-bionic leg to runtime-extra-platforms #74621

Merged

ulisesh added the Known Build Error Use this to report build issues in the .NET Helix tab label Aug 29, 2022

steveisok added a commit that referenced this issue Aug 30, 2022

Skip System.Net.Requests tests on mono linux-arm64 (#74668)

d196d85

The suite crashes on mono linux-arm64 and is being tracked by #74667

jkotas added disabled-test The test is disabled in source code against the issue and removed Known Build Error Use this to report build issues in the .NET Helix tab labels Aug 30, 2022

jeffschwMSFT added area-System.Net and removed area-System.Threading labels Aug 30, 2022

wfurt mentioned this issue Aug 31, 2022

assert in ProcessWaitState on Linux arm64 #74795

Closed

elinor-fung mentioned this issue Sep 2, 2022

Fix testhost build with different libraries and host configurations #74903

Merged

akoeplinger mentioned this issue Sep 2, 2022

System.Net.Requests: Pick correct RemoteExecutor overload in test #74994

Merged

ghost added the in-pr There is an active PR which will close this issue when it is merged label Sep 2, 2022

akoeplinger closed this as completed in #74994 Sep 4, 2022

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Sep 4, 2022

ghost locked as resolved and limited conversation to collaborators Oct 5, 2022

karelz added this to the 8.0.0 milestone Mar 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mono crash in System.Net.Requests tests on linux-arm64 #74667

Mono crash in System.Net.Requests tests on linux-arm64 #74667

steveisok commented Aug 26, 2022 •

edited by jkotas

Loading

ghost commented Aug 26, 2022

ghost commented Aug 30, 2022

Report

Summary

wfurt commented Aug 30, 2022

steveisok commented Aug 30, 2022

wfurt commented Aug 30, 2022

steveisok commented Aug 30, 2022

akoeplinger commented Aug 31, 2022

wfurt commented Aug 31, 2022

karelz commented Sep 1, 2022

akoeplinger commented Sep 5, 2022

Mono crash in System.Net.Requests tests on linux-arm64 #74667

Mono crash in System.Net.Requests tests on linux-arm64 #74667

Comments

steveisok commented Aug 26, 2022 • edited by jkotas Loading

Report

Summary

ghost commented Aug 26, 2022

ghost commented Aug 30, 2022

Report

Summary

wfurt commented Aug 30, 2022

steveisok commented Aug 30, 2022

wfurt commented Aug 30, 2022

steveisok commented Aug 30, 2022

akoeplinger commented Aug 31, 2022

wfurt commented Aug 31, 2022

karelz commented Sep 1, 2022

akoeplinger commented Sep 5, 2022

steveisok commented Aug 26, 2022 •

edited by jkotas

Loading