Known samples for AndroidEnableMarshalMethods=true #8253

jonathanpeppers · 2023-08-08T13:55:21Z

Android application type

.NET Android (net7.0-android, etc.)

Affected platform version

.NET 8 Preview 6

Description

We have turned off AndroidEnableMarshalMethods for .NET 8 due to too many issues surfaced during public testing. We plan on resolving those issues so it can be turned on for .NET 9.

This is just an issue of known examples that fail at runtime when AndroidEnableMarshalMethods=true:

Release crash on unmanagedcallersonly framework code #7876
[Marshal Methods] MAUI project infinite loop on startup #7893
App becomes unresponsive when using plugin.fingerprint on net8-pre5 #8147
ANR with MediaPlayer and AndroidEnableMarshalMethods #9064
[Android, 9.0 Preview 5] Release configuration build results in a malfunctioning app (Preview 5 debug and Preview 4 release configurations work fine). #9115
java.lang.UnsatisfiedLinkError: No implementation found for void crc64953c68cad4e91ae3.GlmApp.n_OnAppForegrounded() - is the library loaded, e.g. System.loadLibrary? #9209
[regression/8.0.0-preview.6.8686] [Android] BitmapFactory.DecodeStreamAsync crashes the app on .NET 8-preview.6 in Release mode maui#16514
Visual Studio 17.7.0 Pre2 / Dotnet 8 8.0.100-preview.5.23303.2 / MAUI 8.0.0-preview.5.8529 build error XAGJS7023 for Android maui#15947
(maybe) [8.0.0-preview.5.8529] [Android] DeviceSensors produce an ANR at irregular intervals when built in Release mode. maui#15829
(maybe) [regression/8.0.0] Android app crashes with an ANR on .NET 8 preview 6 maui#16418

Just linking them here for us to test later in future releases.

Steps to Reproduce

See links above.

Did you find any workaround?

No response

Relevant log output

No response

The text was updated successfully, but these errors were encountered:

tranb3r · 2024-07-23T11:00:57Z

@grendello
Why did you enable marshal methods by default in net9-pre6? #8925
Did you test the known samples before releasing this feature again?
My sample #8147 is broken in net9-pre6, and once again, I have to manually disable marshal methods.

grendello · 2024-07-23T12:00:30Z

@tranb3r This is a very useful feature which gives a noticeable performance increase for huge majority of apps. The only known bad scenario we had was with Blazor apps, and those disable marshal methods by default. We enabled them in preview precisely for the reason of finding what else breaks and add it to the "disable" list if we can identify the issue. Note that marshal methods themselves are fine, the bug doesn't lie in their implementation as far as we can tell (we've had them around for nearly two years, they essentially implement in native code what is implemented in managed code when they are disabled) - the sole fact that they are that much faster triggers a race condition (at least in Blazor) of some sort. We haven't been able to determine what exactly trips up Blazor. If your sample has a pattern we can identify, we'd add it to the "disabled" list.

For the gain in 90% (or more) of apps out there, it is worth enabling the feature by default and disable it explicitly (or automatically, if the pattern is identified as explained above) for the remaining 10% (or less) apps.

grendello · 2024-07-23T12:02:38Z

@tranb3r as far as testing is concerned, we have hundreds of thousands of tests that run on CI and none of them fail with marshal methods. Without enabling the feature by default, we won't ever be able to gather information about what doesn't work "out there".

grendello · 2024-07-23T12:04:57Z

@tranb3r have you seen the problem with anything other than Plugin.Fingerprint?

tranb3r · 2024-07-23T12:11:52Z

For the gain in 90% (or more) of apps out there, it is worth enabling the feature by default and disable it explicitly (or automatically, if the pattern is identified as explained above) for the remaining 10% (or less) apps.

Sure.
However, considering I'm facing at least 2 different issues in my app, caused by this feature, I'm a bit concerned.
Race condition usually occur with complex scenarios, that are difficult to reproduce and integrate in CI.

As far as testing is concerned, we have hundreds of thousands of tests that run on CI and none of them fail with marshal methods.

This is why I'm asking if the samples in this issue have been tested.
My sample is failing, so I suspect this one at least hasn't been investigated/tested.

Have you seen the problem with anything other than Plugin.Fingerprint?

Yes. ANR when doing http requests. However I haven't been able to create a simple repro for it.

bcaceiro · 2024-07-23T12:33:50Z

This option should only be used in .net9 previews right? It is supposed to be somewhat broken on .net8? Because I have tried in the previous maui - sr release and also had some issues, and disabled it.

grendello · 2024-07-23T15:05:50Z

@bcaceiro it was present in net8 but was off by default, correct.

grendello · 2024-07-23T15:13:13Z

For the gain in 90% (or more) of apps out there, it is worth enabling the feature by default and disable it explicitly (or automatically, if the pattern is identified as explained above) for the remaining 10% (or less) apps.

Sure. However, considering I'm facing at least 2 different issues in my app, caused by this feature, I'm a bit concerned. Race condition usually occur with complex scenarios, that are difficult to reproduce and integrate in CI.

Superficial simplicity is unfortunately par for the course in a lot of .NET. One example of that is Blazor apps, another is async/await or LINQ. There is an awful lot of asynchronous or simply complicated code in those features that can fail under some circumstances without any obvious signals as to what's going on. Your sample uses an external service which definitely can introduce threading and/or race condition scenarios. That said, it's much, much simpler than the Blazor "hello world" app that we were investigating. I need to bring some tools up to date and hopefully your sample will give a clue as to what's failing and why.

As far as testing is concerned, we have hundreds of thousands of tests that run on CI and none of them fail with marshal methods.

This is why I'm asking if the samples in this issue have been tested. My sample is failing, so I suspect this one at least hasn't been investigated/tested.

The sad truth is that we don't have much time for manual testing. However, product previews are exactly where we can hope people like you, early testers, will discover issues and let us know about them. Alas, there are very few people following your footsteps and sometimes we learn about issues very late in the game. If we discover that marshal methods cause more trouble than we thought, we're going to disable them by default before the release and hopefully re-enable for .NET 10 - the feature is too valuable to just drop it (and we have more optimizations in mind that build on top of it).

Have you seen the problem with anything other than Plugin.Fingerprint?

Yes. ANR when doing http requests. However I haven't been able to create a simple repro for it.

That's another scenario involving threads and synchronization, open for race conditions. Hopefully all of them are caused by the same issue and maybe your sample app will help us pin point the problem and swat it for good.

filipnavara · 2024-09-28T15:45:22Z

Here's a significantly reduced repro for the deadlock: https://github.com/filipnavara/mm-deadlock-repro. There's no MAUI or Blazor, just 50ish lines of .NET Android code in an empty template.

Essentially it boils down to the .NET GC being dead-locked / live-locked in the stop-the-world stage. I have seen several patterns of this.

Just run it in Release mode and it will immediately freeze before getting to display the "Hello" page.

Once the app is in frozen state, the main thread is waiting in Android code in MediaPlayer.Prepare(). Meanwhile, the media player loads the data in separate thread which calls back StreamMediaDataSource.ReadAt. This intentionally allocates empty arrays to trigger GCs to make the repro deterministic. In real world this allocation will happen at the marshalling level for byte[] arrays and trigger the same code path. The GC will deterministically freeze in the stop-the-world stage. It seems to wait for one of the threads, either the one executing MediaPlayer.Prepare() (less likely) or a random Java thread pool thread not running any .NET code at the moment (more likely).

In an unreduced repro, when running under debugger, there was a native exception in art_quick_generic_jni_trampoline prior to getting into the broken state. This may have caused some corruption that ultimately puts the Mono thread list in inconsistent state. I haven't been able to verify if this also happens in the reduced repro due to some struggle with LLDB.

UPD: Here’s a log with all the thread state transitions in MonoVM: https://gist.github.com/filipnavara/1f2e220063022cefe1375b7c87efa1ee. Seems like some code is initially run on one of the Java thread pool threads and that ends up with a wrong state. Next .NET GC from another Java thread pool thread fails to stop the world.

UPD2: Fixed trace with names in MONO_STACKDATA: https://gist.github.com/filipnavara/fd997e3e9879c072935bde866ef51aa8

filipnavara · 2024-09-29T07:42:52Z

I have narrowed down the thread state transition issue to this:

09-29 09:38:40.655  7970  7985 E THREAD  : attaching 0xb4000073ec62c9a0
09-29 09:38:40.655  7970  7985 E THREAD  : registering info 0xb4000073ec62c9a0 tid 0x726597f730 small id 43
09-29 09:38:40.655  7970  7985 E THREAD  : [ATTACH][0x726597f730] STARTING . -> RUNNING . (0 -> 0) 
09-29 09:38:40.655  7970  7985 E THREAD  : [DO_BLOCKING][0x726597f730] RUNNING . -> STATE_BLOCKING . (0 -> 0) mono_thread_info_suspend_lock_with_info
09-29 09:38:40.655  7970  7985 E THREAD  : [DONE_BLOCKING][0x726597f730] STATE_BLOCKING . -> RUNNING . (0 -> 0) mono_thread_info_suspend_lock_with_info
09-29 09:38:40.655  7970  7985 E THREAD  : [ABORT_BLOCKING][0x726597f730] RUNNING . -> RUNNING . (0 -> 0) mono_class_get
09-29 09:38:40.656  7970  7985 E THREAD  : [ABORT_BLOCKING][0x726597f730] RUNNING . -> RUNNING . (0 -> 0) mono_method_get_unmanaged_callers_only_ftnptr
09-29 09:38:40.656  7970  7985 E THREAD  : [ABORT_BLOCKING][0x726597f730] RUNNING . -> RUNNING . (0 -> 0) mono_threads_attach_coop
09-29 09:38:40.656  7970  7985 E THREAD  : XX: mono_threads_attach_coop_internal external 0 cookie 0x0

At that point the thread pool thread is attached to the runtime. Yet the very next call to mono_threads_attach_coop_internal, ie. entering the UnmanagedCallersOnly method, fails to identify the thread as external. This causes the transition cookie to be NULL, and exiting the call with mono_threads_detach_coop becomes no-op which leaves the thread in a wrong state.

filipnavara · 2024-09-29T07:49:28Z

The sequence above is result of this:

android/src/native/monodroid/xamarin-android-app-context.cc

Lines 60 to 71 in 3fbc451

    
           // We need to do that, as Mono APIs cannot be invoked from threads that aren't attached to the runtime. 
        
           mono_thread_attach (mono_get_root_domain ()); 
        
           MonoImage *image = MonoImageLoader::get_from_index (mono_image_index); 
        
           MarshalMethodsManagedClass &klass = marshal_methods_class_cache[class_index]; 
        
           if (klass.klass == nullptr) { 
        
           	klass.klass = image != nullptr ? mono_class_get (image, klass.token) : nullptr; 
        
           } 
        
           MonoMethod *method = klass.klass != nullptr ? mono_get_method (image, method_token, klass.klass) : nullptr; 
        
           MonoError error; 
        
           void *ret = method != nullptr ? mono_method_get_unmanaged_callers_only_ftnptr (method, &error) : nullptr;

get_function_pointer knowingly calls mono_thread_attach which attaches the thread pool thread to Mono runtime. It never detaches it, which leads to an incorrect state of the thread pool threads.

filipnavara · 2024-09-29T11:02:56Z

So, the simple fix is to replace mono_thread_attach with the deprecated mono_jit_thread_attach API. Unfortunately, mono_threads_attach_coop/mono_threads_detach_coop was never added to the public API headers and neither was mono_threads_enter_gc_unsafe_region/mono_threads_exit_gc_unsafe_region. These would be the proper way to fix it, I think.

Notably, mono_jit_thread_attach is still the way used by xamarin-macios.

cc @lambdageek

Context: #8253 Context: #9343 We've had a number of reports about *hangs* when LLVM Marshal Methods are enabled (see also #8253), but we had been willing to accept having LLVM Marshal Methods enabled by default as it helps improve app startup time. However, [a customer reported a native crash][0] when LLVM Marshal Methods are enabled: > ![Thread Stack Trace][1] syscall at line 28 within libc __futex_wait_ex(void volatile*, bool, int, bool, timespec const*) at line 144 within libc sem_wait at line 108 within libc within libmonosgen-2.0.so (Build Id: …) within <anonymous:…> This native crash suggests "something going wrong" within MonoVM. We find this "scary" enough that we feel it is once again prudent to disable LLVM Marshal Methods by default in .NET 9, by setting the default value of `$(AndroidEnableMarshalMethods)`=False. Even though #9343 has a potential fix for the hang, we feel that we're too close to .NET 9 GA to validate in the wild. This feature will need to wait for .NET 10. 🙁 To explicitly enable LLVM Marshal Methods, set `$(AndroidEnableMarshalMethods)`=True in your App `.csproj`. [0]: https://discord.com/channels/732297728826277939/732297837953679412/1288758900627345463 [1]: https://cdn.discordapp.com/attachments/732297837953679412/1288758900522483712/05F6ED48-2BBA-4FC0-A54F-81BB57606500.png?ex=66f659c1&is=66f50841&hm=8f6f78f283f8c03e9fb37a452ad5d0babaaad3dd4bcc1b4887f4dcf60f651c12&

Context: dotnet#8253 Context: dotnet#9343 We've had a number of reports about *hangs* when LLVM Marshal Methods are enabled (see also dotnet#8253), but we had been willing to accept having LLVM Marshal Methods enabled by default as it helps improve app startup time. However, [a customer reported a native crash][0] when LLVM Marshal Methods are enabled: > ![Thread Stack Trace][1] syscall at line 28 within libc __futex_wait_ex(void volatile*, bool, int, bool, timespec const*) at line 144 within libc sem_wait at line 108 within libc within libmonosgen-2.0.so (Build Id: …) within <anonymous:…> This native crash suggests "something going wrong" within MonoVM. We find this "scary" enough that we feel it is once again prudent to disable LLVM Marshal Methods by default in .NET 9, by setting the default value of `$(AndroidEnableMarshalMethods)`=False. Even though dotnet#9343 has a potential fix for the hang, we feel that we're too close to .NET 9 GA to validate in the wild. This feature will need to wait for .NET 10. 🙁 To explicitly enable LLVM Marshal Methods, set `$(AndroidEnableMarshalMethods)`=True in your App `.csproj`. [0]: https://discord.com/channels/732297728826277939/732297837953679412/1288758900627345463 [1]: https://cdn.discordapp.com/attachments/732297837953679412/1288758900522483712/05F6ED48-2BBA-4FC0-A54F-81BB57606500.png?ex=66f659c1&is=66f50841&hm=8f6f78f283f8c03e9fb37a452ad5d0babaaad3dd4bcc1b4887f4dcf60f651c12&

…9346) Context: #8253 Context: #9343 We've had a number of reports about *hangs* when LLVM Marshal Methods are enabled (see also #8253), but we had been willing to accept having LLVM Marshal Methods enabled by default as it helps improve app startup time. However, [a customer reported a native crash][0] when LLVM Marshal Methods are enabled: > ![Thread Stack Trace][1] syscall at line 28 within libc __futex_wait_ex(void volatile*, bool, int, bool, timespec const*) at line 144 within libc sem_wait at line 108 within libc within libmonosgen-2.0.so (Build Id: …) within <anonymous:…> This native crash suggests "something going wrong" within MonoVM. We find this "scary" enough that we feel it is once again prudent to disable LLVM Marshal Methods by default in .NET 9, by setting the default value of `$(AndroidEnableMarshalMethods)`=False. Even though #9343 has a potential fix for the hang, we feel that we're too close to .NET 9 GA to validate in the wild. This feature will need to wait for .NET 10. 🙁 To explicitly enable LLVM Marshal Methods, set `$(AndroidEnableMarshalMethods)`=True in your App `.csproj`. [0]: https://discord.com/channels/732297728826277939/732297837953679412/1288758900627345463 [1]: https://cdn.discordapp.com/attachments/732297837953679412/1288758900522483712/05F6ED48-2BBA-4FC0-A54F-81BB57606500.png?ex=66f659c1&is=66f50841&hm=8f6f78f283f8c03e9fb37a452ad5d0babaaad3dd4bcc1b4887f4dcf60f651c12& Co-authored-by: Marek Habersack <grendel@twistedcode.net>

) Fixes: #8253 `get_function_pointer` can be called from any thread. In particular, it can be called from Android thread pool thread when method marshaling is used. We need to ensure those threads stay in the GC safe mode from the runtime perspective once we stop executing user code on those threads. `mono_thread_attach` switches the thread to GC unsafe mode and it stays there after we stop executing user code. The next GC will try to suspend the thread which cannot succeed since there's no cooperative runtime code running on that thread anymore. The fix is to use `mono_jit_thread_attach` which mimics what the iOS/macOS interop does. It creates the necessary MonoVM thread structures for JIT to work but doesn't transition the thread state.

Context: #9343 Context: #8253 (comment) Test is based on the original repro from https://github.com/filipnavara/mm-deadlock-repro/

jonathanpeppers added Area: App+Library Build Issues when building Library projects or Application projects. Area: App Runtime Issues in `libmonodroid.so`. labels Aug 8, 2023

jonathanpeppers added this to the .NET 9 Planning milestone Aug 8, 2023

jonathanpeppers assigned grendello and jonathanpeppers Aug 8, 2023

microsoft-github-policy-service bot added the needs-triage Issues that need to be assigned. label Aug 8, 2023

jonathanpeppers removed the needs-triage Issues that need to be assigned. label Aug 8, 2023

jonathanpeppers mentioned this issue Aug 30, 2023

[regression/8.0.0] Android app crashes with an ANR on .NET 8 preview 6 dotnet/maui#16418

Closed

jpobst added the bug Component does not function as intended. label Oct 10, 2023

This was referenced May 17, 2024

App becomes unresponsive when using plugin.fingerprint on net8-pre5 #8147

Closed

[Marshal Methods] MAUI project infinite loop on startup #7893

Closed

Release crash on unmanagedcallersonly framework code #7876

Closed

kerams mentioned this issue Jun 28, 2024

ANR with MediaPlayer and AndroidEnableMarshalMethods #9064

Open

This was referenced Sep 26, 2024

java.lang.UnsatisfiedLinkError: No implementation found for void crc64953c68cad4e91ae3.GlmApp.n_OnAppForegrounded() - is the library loaded, e.g. System.loadLibrary? #9209

Open

IL Linker Intrinsics + Marshal Methods, oh my! #8155

Closed

filipnavara mentioned this issue Sep 29, 2024

Fix thread state transition in get_function_pointer #9343

Merged

jonathanpeppers mentioned this issue Sep 30, 2024

[release/9.0.1xx-rc2] [Xamarin.Android.Build.Tasks] Disable LLVM marshal methods (#9336) #9346

Merged

jonathanpeppers closed this as completed in #9343 Sep 30, 2024

grendello added a commit that referenced this issue Oct 1, 2024

Add test for marshal methods hang

6a99773

Context: #9343 Context: #8253 (comment) Test is based on the original repro from https://github.com/filipnavara/mm-deadlock-repro/

grendello mentioned this issue Oct 1, 2024

Add test for marshal methods hang #9352

Merged

dellis1972 pushed a commit that referenced this issue Oct 2, 2024

Add test for marshal methods hang (#9352)

ba0a61f

Context: #9343 Context: #8253 (comment) Test is based on the original repro from https://github.com/filipnavara/mm-deadlock-repro/

grendello mentioned this issue Oct 4, 2024

Increased amount of ANRs after disabling concurrent GC #9365

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Known samples for AndroidEnableMarshalMethods=true #8253

Known samples for AndroidEnableMarshalMethods=true #8253

jonathanpeppers commented Aug 8, 2023 •

edited by jonpryor

Loading

tranb3r commented Jul 23, 2024

grendello commented Jul 23, 2024

grendello commented Jul 23, 2024

grendello commented Jul 23, 2024

tranb3r commented Jul 23, 2024 •

edited

Loading

bcaceiro commented Jul 23, 2024

grendello commented Jul 23, 2024

grendello commented Jul 23, 2024 •

edited

Loading

filipnavara commented Sep 28, 2024 •

edited

Loading

filipnavara commented Sep 29, 2024

filipnavara commented Sep 29, 2024

filipnavara commented Sep 29, 2024

Known samples for AndroidEnableMarshalMethods=true #8253

Known samples for AndroidEnableMarshalMethods=true #8253

Comments

jonathanpeppers commented Aug 8, 2023 • edited by jonpryor Loading

Android application type

Affected platform version

Description

Steps to Reproduce

Did you find any workaround?

Relevant log output

tranb3r commented Jul 23, 2024

grendello commented Jul 23, 2024

grendello commented Jul 23, 2024

grendello commented Jul 23, 2024

tranb3r commented Jul 23, 2024 • edited Loading

bcaceiro commented Jul 23, 2024

grendello commented Jul 23, 2024

grendello commented Jul 23, 2024 • edited Loading

filipnavara commented Sep 28, 2024 • edited Loading

filipnavara commented Sep 29, 2024

filipnavara commented Sep 29, 2024

filipnavara commented Sep 29, 2024

jonathanpeppers commented Aug 8, 2023 •

edited by jonpryor

Loading

tranb3r commented Jul 23, 2024 •

edited

Loading

grendello commented Jul 23, 2024 •

edited

Loading

filipnavara commented Sep 28, 2024 •

edited

Loading