Microsoft.Data.SqlClient is not compatible with AssemblyLoadContext unloading

[AlbertoCe]

Hello,

when I load a plugin that uses EntityFrameworkCore (using a custom collectible AssemblyLoadContext) and try to unload it, it does not get unloaded: the IsActive property of the context's weakreference remain true even after multiple GC.Collect() and GC.WaitForPendingFinalizers().

It works well if I comment out all the EntityFrameworkCore related code from the plugin.

[AlbertoCe]

This is the GCRoot dump of the LoaderAllocator instance:

0:013> !gcroot -all 000001ddd2dbad28
HandleTable:
    000001DDD11C1320 (strong handle)
    -> 000001DDD2DC7B18 System.Object[]
    -> 000001DDD2DD0DC8 System.Threading.Thread
    -> 000001DDD2EBDF10 System.Threading.ExecutionContext
    -> 000001DDD2EBDEF0 System.Threading.AsyncLocalValueMap+OneElementAsyncLocalValueMap
    -> 000001DDD2EBDAC0 System.Threading.AsyncLocal`1[[Microsoft.Extensions.Caching.Memory.CacheEntryStack, Microsoft.Extensions.Caching.Memory]]
    -> 000001DDD2DBAD28 System.Reflection.LoaderAllocator

    000001DDD11C13F8 (strong handle)
    -> 000001DDD2DD0DC8 System.Threading.Thread
    -> 000001DDD2EBDF10 System.Threading.ExecutionContext
    -> 000001DDD2EBDEF0 System.Threading.AsyncLocalValueMap+OneElementAsyncLocalValueMap
    -> 000001DDD2EBDAC0 System.Threading.AsyncLocal`1[[Microsoft.Extensions.Caching.Memory.CacheEntryStack, Microsoft.Extensions.Caching.Memory]]
    -> 000001DDD2DBAD28 System.Reflection.LoaderAllocator

    000001DDD11C2700 (strong handle)
    -> 000001DDD2EFD2D8 Microsoft.Data.SqlClient.TdsParserStateObjectNative
    -> 000001DDD2DBAD28 System.Reflection.LoaderAllocator

    000001DDD11C15D8 (pinned handle)
    -> 000001DDE2DB54A0 System.Object[]
    -> 000001DDD2DE2788 System.Dynamic.Utils.CacheDict`2[[System.Reflection.MethodBase, System.Private.CoreLib],[System.Reflection.ParameterInfo[], System.Private.CoreLib]]
    -> 000001DDD2DE27A8 System.Dynamic.Utils.CacheDict`2+Entry[[System.Reflection.MethodBase, System.Private.CoreLib],[System.Reflection.ParameterInfo[], System.Private.CoreLib]][]
    -> 000001DDD2E4F490 System.Dynamic.Utils.CacheDict`2+Entry[[System.Reflection.MethodBase, System.Private.CoreLib],[System.Reflection.ParameterInfo[], System.Private.CoreLib]]
    -> 000001DDD2E4F2F0 System.Reflection.RuntimeMethodInfo
    -> 000001DDD2DBAD28 System.Reflection.LoaderAllocator

    000001DDD11C15F8 (pinned handle)
    -> 000001DDE2DB1038 System.Object[]
    -> 000001DDD2EC27B0 System.Threading.TimerQueue[]
    -> 000001DDD2EC2808 System.Threading.TimerQueue
    -> 000001DDD2EDD6E0 System.Threading.TimerQueueTimer
    -> 000001DDD2EC2750 System.Threading.TimerQueueTimer
    -> 000001DDD2EC2690 System.Threading.TimerCallback
    -> 000001DDD2EC2510 Microsoft.Data.SqlClient.SqlConnectionFactory
    -> 000001DDD2DBAD28 System.Reflection.LoaderAllocator

Found 5 roots.

[ericstj]

@janvorli this looks similar to other problems you helped resolve around this scenario.

[janvorli]

So, there are several things that holds on objects from the unloadable context.

• There are two AsyncLocal<Microsoft.Extensions.Caching.Memory.CacheEntryStack> instances. These are used to capture locals in async function frames for as long as the frame is alive. So it looks there is async code running that has frames with those AsyncLocals on the "async call stack".
• There is a strong GC handle referencing an instance of the Microsoft.Data.SqlClient.TdsParserStateObjectNative that was loaded into the unloadable context.
• There is a System.Reflection.RuntimeMethodInfo instance in the unloadable context being held in System.Dynamic.Utils.CacheDict in the default context.
• There is a queued TimerCallback referencign a method in the Microsoft.Data.SqlClient.SqlConnectionFactory that is loaded into the unloadable context.

[janvorli]

One thought I have is why the Microsoft.Data.SqlClient and System.Dynamic.Runtime assemblies are loaded into the unloadable context. @AlbertoCe how does your custom AssemblyLoadContext resolve assembly references? Does it use the AssemblyDependencyResolver?

[AlbertoCe]

Hello @janvorli ,

I used the sample [https://github.com/dotnet/samples/tree/master/core/tutorials/Unloading].
This is the LoadContext I use:

class HostAssemblyLoadContext : AssemblyLoadContext
    {
        private AssemblyDependencyResolver _resolver;

        public HostAssemblyLoadContext(string pluginPath) : base(isCollectible: true)
        {
            _resolver = new AssemblyDependencyResolver(pluginPath);
        }
        protected override Assembly Load(AssemblyName name)
        {
            Assembly result = null;

                string assemblyPath = _resolver.ResolveAssemblyToPath(name);
                if (assemblyPath != null)
                {
                    result = LoadFromAssemblyPath(assemblyPath);
                }
            return result;
        }

        protected override IntPtr LoadUnmanagedDll(string unmanagedDllName)
        {
                string libraryPath = _resolver.ResolveUnmanagedDllToPath(unmanagedDllName);
                if (libraryPath != null)
                {
                    return LoadUnmanagedDllFromPath(libraryPath);
                }
            return IntPtr.Zero;
        }
    }

[AlbertoCe]

Hello @janvorli ,
Any news on this ?

[janvorli]

@AlbertoCe I am sorry for the delayed response. So, it is clear why the Microsoft.Data.SqlClient was loaded into the unloadable context - I have not realized it is not a shared framework assembly before. And the System.Dynamic.Runtime is actually not loaded into the context, I've said it by mistake.

Regarding the Microsoft.Data.SqlClient, the current implementation is not compatible with unloadability, since it creates the timer callback and the strong GC handle. I have looked into it a bit more when I was answering questions on it in another related issue (https://github.com/dotnet/coreclr/issues/27936). It seems the component can be fixed easily to support unloadability by registering for AssemblyLoadContext.Unload event. Until that happens, it seems there might be a workaround that I've mentioned in the other issue.
However, I have not tried to check if the issue with the RuntimeMethodInfo being stored in the System.Dynamic.Utils.CacheDict can be worked around in a similar manner.

[joperezr]

@janvorli Is there anything we want to fix around this for 5.0 or is it ok to move to future?

[ghost]

Tagging subscribers to this area: @vitek-karas, @agocke
See info in area-owners.md if you want to be subscribed.

[danmoseley]

Leaving in 5.0 so loader folks can triage

[agocke]

Sounds like this is an issue with Microsoft.Data.SqlClient. Created an issue at dotnet/SqlClient#1687. Closing this one.