Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System.Diagnostics.TraceInternal.Listeners.get timed out #55696

Closed
TimothyMothra opened this issue Jul 15, 2021 · 16 comments
Closed

System.Diagnostics.TraceInternal.Listeners.get timed out #55696

TimothyMothra opened this issue Jul 15, 2021 · 16 comments

Comments

@TimothyMothra
Copy link

TimothyMothra commented Jul 15, 2021

I represent the Application Insights .NET SDK.
Our customers are reporting an issue with our TraceListener.

image

Related source: https://referencesource.microsoft.com/#System/compmod/system/diagnostics/TraceInternal.cs,30

Description

Per our investigation, we're seeing deadlocks involving clr native code and System.Diagnostics.TraceInternal and need help investigating.

Related issues:

Reproduction

We had a customer kindly share a repro here:

Investigation

We investigated using WinDbg and found evidence of deadlocks.

DEADLOCK DETECTED
Thread 49 holds CriticalSection 23016cd8
...and is waiting for the lock on SyncBlock 07bc55ac OBJ:0e7ea3f4[System.Collections.Generic.List1[[System.WeakReference, mscorlib]]] Thread 5 holds the lock on SyncBlock 07bc55ac OBJ:0e7ea3f4[System.Collections.Generic.List1[[System.WeakReference, mscorlib]]]
...and is waiting on CriticalSection 23016cd8
Thread 49 is waiting at System.Diagnostics.Tracing.EventListener.AddEventSource(System.Diagnostics.Tracing.EventSource)(+50 IL,+0x42 Native)
Thread 5 is waiting at clr!CrstBase::SpinEnter+0xd6

Thread 49 => clr code => Entered critsec => Executing code => Managed code => Waiting to enter a managed lock

0:049> kL
# ChildEBP RetAddr      
00 23bbe080 76f3b143     ntdll!NtWaitForMultipleObjects+0xc
01 23bbe080 6dc54541     KERNELBASE!WaitForMultipleObjectsEx+0x103
02 23bbe0d0 6dc54190     clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x3c
03 (Inline) --------     clr!Thread::DoAppropriateAptStateWait+0x35
04 23bbe154 6dc542be     clr!Thread::DoAppropriateWaitWorker+0x1eb
05 23bbe1c0 6dc54407     clr!Thread::DoAppropriateWait+0x64
06 (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException+0xfffffffe
07 23bbe20c 6db633db     clr!CLREventBase::WaitEx+0x121
08 23bbe220 6dc3f29b     clr!CLREventBase::Wait+0x1a
09 23bbe2ac 6dc3f3cc     clr!AwareLock::EnterEpilogHelper+0xa8
0a (Inline) --------     clr!ThreadDebugBlockingInfo::PopBlockingItem+0xfffffff9
0b (Inline) --------     clr!DebugBlockingItemHolder::{dtor}+0xfffffff9
0c 23bbe2f4 6dc3f1a5     clr!AwareLock::EnterEpilog+0x48
0d 23bbe3b8 6dc71152     clr!AwareLock::Enter+0x4a
0e (Inline) --------     clr!SyncBlock::EnterMonitor+0x7
0f (Inline) --------     clr!ObjHeader::EnterObjMonitor+0x12
10 (Inline) --------     clr!Object::EnterObjMonitor+0x18
11 23bbe3b8 6ca6b012     clr!JITutil_MonReliableEnter+0xb5
12 23bbe3f0 6ca6a753     mscorlib_ni!System.Diagnostics.Tracing.EventListener.AddEventSource+0x42
13 23bbe424 6ca4d87b     mscorlib_ni!System.Diagnostics.Tracing.EventSource.Initialize+0x11f
14 23bbe458 21c68317     mscorlib_ni!System.Diagnostics.Tracing.EventSource..ctor+0x4f
15 23bbe46c 21c682b7     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.CoreEventSource..ctor+0x37
16 23bbe478 6db4f066     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.CoreEventSource..cctor+0x1f
17 23bbe484 6db5234a     clr!CallDescrWorkerInternal+0x34
18 23bbe4d8 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
19 23bbe51c 6db64392     clr!DispatchCallDebuggerWrapper+0x59
1a 23bbe568 6dc54fde     clr!DispatchCallSimple+0x97
1b 23bbe5d4 6dc550ab     clr!MethodTable::RunClassInitEx+0xca
1c 23bbeab8 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x45d
1d (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
1e 23bbeb40 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
1f 23bbeb74 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
WARNING: Frame IP not in any known module. Following frames may be wrong.
20 23bbeb74 21c67ff7     0x83e34b1
21 23bbebb4 21c67e3c     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.DequeueAndSend+0x5f
22 23bbebe4 6cabd4bb     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Runner+0x54
23 23bbebf0 6cabb731     mscorlib_ni!System.Threading.Tasks.Task.InnerInvoke+0x4b
24 23bbec14 6cabb6fc     mscorlib_ni!System.Threading.Tasks.Task.Execute+0x31
25 23bbec7c 6ca58604     mscorlib_ni!System.Threading.Tasks.Task.ExecutionContextCallback+0x1c
26 23bbec7c 6ca58537     mscorlib_ni!System.Threading.ExecutionContext.RunInternal+0xc4
27 23bbec90 6cabb4b2     mscorlib_ni!System.Threading.ExecutionContext.Run+0x17
28 23bbecfc 6cabb357     mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0xe2
29 23bbed0c 6cad6d0a     mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry+0xb7
2a 23bbed1c 6ca32dd6     mscorlib_ni!System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork+0x1e
2b 23bbed1c 6ca58604     mscorlib_ni!System.Threading.ThreadHelper.ThreadStart_Context+0x76
2c 23bbed80 6ca58537     mscorlib_ni!System.Threading.ExecutionContext.RunInternal+0xc4
2d 23bbed94 6ca584f4     mscorlib_ni!System.Threading.ExecutionContext.Run+0x17
2e 23bbedb0 6ca32d0e     mscorlib_ni!System.Threading.ExecutionContext.Run+0x44
2f 23bbedc8 6db4f066     mscorlib_ni!System.Threading.ThreadHelper.ThreadStart+0x52
30 23bbedd4 6db5234a     clr!CallDescrWorkerInternal+0x34
31 23bbee28 6db5862b     clr!CallDescrWorkerWithHandler+0x6b
32 23bbee9c 6dc69727     clr!MethodDescCallSite::CallTargetWorker+0x16a
33 (Inline) --------     clr!MethodDescCallSite::Call+0x12
34 23bbf00c 6dcfe376     clr!ThreadNative::KickOffThread_Worker+0x131
35 23bbf020 6dcfe401     clr!ManagedThreadBase_DispatchInner+0x71
36 (Inline) --------     clr!ManagedThreadBase_DispatchMiddle::__l16::Cleanup::{dtor}+0xfffffffc
37 23bbf0c4 6dcfe2f2     clr!ManagedThreadBase_DispatchMiddle+0x7e
38 23bbf118 6dd01655     clr!ManagedThreadBase_DispatchOuter+0x99
39 23bbf124 6dd015e1     clr!ManagedThreadBase_DispatchInCorrectAD+0x15
3a 23bbf1f0 6dd01680     clr!Thread::DoADCallBack+0x328
3b 23bbf214 6dcfe401     clr!ManagedThreadBase_DispatchInner+0x4e
3c (Inline) --------     clr!ManagedThreadBase_DispatchMiddle::__l16::Cleanup::{dtor}+0xfffffffc
3d 23bbf2b8 6dcfe2f2     clr!ManagedThreadBase_DispatchMiddle+0x7e
3e 23bbf30c 6dcfe4e1     clr!ManagedThreadBase_DispatchOuter+0x99
3f 23bbf330 6dc695d8     clr!ManagedThreadBase_FullTransitionWithAD+0x2f
40 23bbf3b4 6dc14c87     clr!ThreadNative::KickOffThread+0x260
41 23bbf9d8 7558fa29     clr!Thread::intermediateThreadProc+0x58
42 23bbf9e8 77497a7e     kernel32!BaseThreadInitThunk+0x19
43 23bbfa44 77497a4e     ntdll!__RtlUserThreadStart+0x2f
44 23bbfa54 00000000     ntdll!_RtlUserThreadStart+0x1b

Thread 5 => clr code => managed code => Entered lock => executing code => clr code => Waiting to enter critsec

0:005> kL
# ChildEBP RetAddr      
00 071d9c58 7748f909     ntdll!NtWaitForAlertByThreadId+0xc
01 071d9c58 7748f65d     ntdll!RtlpWaitOnAddressWithTimeout+0x64
02 (Inline) --------     ntdll!RtlpWaitOnAddress+0xbb
03 071d9cf8 774700ca     ntdll!RtlpWaitOnCriticalSection+0x18d
04 071d9d30 7746ff19     ntdll!RtlpEnterCriticalSectionContended+0x1aa
05 071d9d3c 6dc5b360     ntdll!RtlEnterCriticalSection+0x49
06 (Inline) --------     clr!UnsafeEnterCriticalSection+0x7
07 071d9d74 6dc5b2bb     clr!CrstBase::SpinEnter+0xd6
08 071d9d74 6db64b9c     clr!CrstBase::Enter+0x136
09 (Inline) --------     clr!Volatile<Thread *>::{ctor}+0xfffffffb
0a (Inline) --------     clr!VolatilePtr<Thread,Thread *>::{ctor}+0xfffffffb
0b (Inline) --------     clr!DeadlockAwareLock::EndEnterLock
0c (Inline) --------     clr!ListLockEntry::FinishDeadlockAwareEnter+0x1e
0d 071d9da4 6dc54e02     clr!ListLockEntry::LockHolder::DeadlockAwareAcquire+0x3e
0e 071da284 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x3e8
0f (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
10 071da30c 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
11 071da33c 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
WARNING: Frame IP not in any known module. Following frames may be wrong.
12 071da33c 21c6ac0d     0x83e34b1
13 071da354 21c6aa42     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingManager<System.__Canon>..ctor+0x6d
14 071da394 21c6a865     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule.Initialize+0xba
15 071da3c0 21c6a90a     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponent+0x35
16 071da3e8 21c6a815     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents+0x42
17 071da3f8 21c68e63     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents+0x4d
18 071da428 21c6851f     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize+0x22b
19 071da440 21c66445     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize+0x27
1a 071da470 21c65297     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.get_Active+0x95
1b 071da484 21c64e5b     Microsoft_ApplicationInsights_TraceListener!Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor+0x27
1c 071da494 6db4f066     Microsoft_ApplicationInsights_TraceListener!Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor+0xb
1d 071da494 6db5234a     clr!CallDescrWorkerInternal+0x34
1e 071da4e8 6db677f1     clr!CallDescrWorkerWithHandler+0x6b
1f 071da528 6db678f6     clr!CallDescrWorkerReflectionWrapper+0x55
20 071da82c 6ca85ce9     clr!RuntimeMethodHandle::InvokeMethod+0x838
21 071da854 2355b9f3     mscorlib_ni!System.Reflection.RuntimeConstructorInfo.Invoke+0x149
22 071da874 2355b421     System!System.SecurityUtils.ConstructorInfoInvoke+0x5b
23 071da8b0 2355b2d8     System!System.Diagnostics.TraceUtils.GetRuntimeObject+0xd9
24 071da8c8 2355afc1     System!System.Diagnostics.TypedElement.BaseGetRuntimeObject+0x30
25 071da8f4 2355a92e     System!System.Diagnostics.ListenerElement.GetRuntimeObject+0x1b1
26 071da928 2355a757     System!System.Diagnostics.ListenerElementsCollection.GetRuntimeObject+0xb6
27 071da958 6db4f066     System!System.Diagnostics.TraceInternal.get_Listeners+0x9f
28 071da964 6db5234a     clr!CallDescrWorkerInternal+0x34
29 071da9b8 6db5862b     clr!CallDescrWorkerWithHandler+0x6b
2a 071daa24 6e0b669c     clr!MethodDescCallSite::CallTargetWorker+0x16a
2b (Inline) --------     clr!MethodDescCallSite::CallWithValueTypes_RetArgSlot+0x6
2c 071daa70 6e0b6ba5     clr!FuncEvalWrapper+0x2e
2d 071dabc4 6e0b6f09     clr!DoNormalFuncEval+0x2b6
2e 071dad24 6e0b7c36     clr!GCProtectArgsAndDoNormalFuncEval+0x201
2f 071daed0 6e0b7dc1     clr!FuncEvalHijackRealWorker+0x796
30 071daf48 6db41186     clr!FuncEvalHijackWorker+0x14d
31 071daf5c 21c64374     clr!FuncEvalHijack+0x6
32 071daf8c 6d2542d2     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource.OnEventCommand+0x24
33 071dafd4 6ca6a986     mscorlib_ni!System.Diagnostics.Tracing.EventSource.DoCommand+0x21e
34 071db008 6ca4d87b     mscorlib_ni!System.Diagnostics.Tracing.EventSource.Initialize+0x352
35 071db03c 21c642e8     mscorlib_ni!System.Diagnostics.Tracing.EventSource..ctor+0x4f
36 071db050 21c64297     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource..ctor+0x28
37 071db050 6db4f066     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource..cctor+0x1f
38 071db05c 6db5234a     clr!CallDescrWorkerInternal+0x34
39 071db0b0 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
3a 071db0f4 6db64392     clr!DispatchCallDebuggerWrapper+0x59
3b 071db140 6dc54fde     clr!DispatchCallSimple+0x97
3c 071db1ac 6dc550ab     clr!MethodTable::RunClassInitEx+0xca
3d 071db690 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x45d
3e (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
3f 071db718 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
40 071db75c 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
41 071db75c 21c6412e     0x83e34b1
42 071db78c 21c6408e     Azure_Identity!+0x36
43 071db78c 21c63529     Azure_Identity!Azure.Core.Pipeline.ClientDiagnostics..ctor+0x4e
44 071db7a4 21c62437     Azure_Identity!+0x71
45 071db7b4 6ca7c49b     Azure_Identity!+0x2f
46 071db7e0 6ca7c10f     mscorlib_ni!System.Lazy`1.CreateValue+0xbb
47 071db814 6ca7c040     mscorlib_ni!System.Lazy`1.LazyInitValue+0x9f
48 071db824 21c61b83     mscorlib_ni!System.Lazy`1.get_Value+0x50
49 071db834 21c621d6     Azure_Identity!+0x43
4a 071db848 21c616cf     Azure_Identity!Azure.Identity.DefaultAzureCredential..ctor+0x26
4b 071db884 21c605ac     Microsoft_Configuration_ConfigurationBuilders_Azure!Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder.LazyInitialize+0x1b7
4c 071db8b4 21c60646     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureInitialized+0x5c
4d 071db8e4 21c60fa0     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.GetValueInternal+0x4e
4e 071db928 21c7686d     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.ProcessConfigurationSection+0xd0
4f 071db930 21c7689b     System_Configuration!System.Configuration.Internal.InternalConfigHost.System.Configuration.Internal.IInternalConfigurationBuilderHost.ProcessConfigurationSection+0x15
50 071db93c 21c76973     System_Configuration!System.Configuration.Internal.DelegatingConfigHost.ProcessConfigurationSection+0x13
51 071db968 21b2f5b8     System_Configuration!System.Configuration.BaseConfigurationRecord.CallHostProcessConfigurationSection+0x33
52 071db984 21b85152     System_Configuration!System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionImpl+0xc8
53 071db9c8 21b2ec7a     System_Configuration!System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionWithRestrictedPermissions+0x62
54 071db9f0 21b2eb24     System_Configuration!System.Configuration.RuntimeConfigurationRecord.CreateSection+0x62
55 071dba3c 21b341b8     System_Configuration!System.Configuration.BaseConfigurationRecord.CallCreateSection+0x6c
56 071dba74 21b33f1f     System_Configuration!System.Configuration.BaseConfigurationRecord.EvaluateOne+0x58
57 071dbb34 21b2df56     System_Configuration!System.Configuration.BaseConfigurationRecord.Evaluate+0x43f
58 071dbbb4 21b2db23     System_Configuration!System.Configuration.BaseConfigurationRecord.GetSectionRecursive+0x416
59 071dbbdc 21bafe86     System_Configuration!System.Configuration.BaseConfigurationRecord.GetSection+0x2b
5a 071dbbec 21c71339     System_Web!System.Web.Util.AppSettings.GetAppSettingsSection+0x56
5b 071dbcf4 21c720a7     System_Web!System.Web.Util.AppSettings.EnsureSettingsLoaded+0x89
5c 071dbd08 21c722b4     System_Web!System.Web.Compilation.BuildManager.RestorePortableCompilationOutputSnapshot+0x1f
5d 071dbd38 085b3098     System_Web!System.Web.Compilation.BuildManager.ExecutePreAppStart+0x14
5e 071dbd70 085b2026     System_Web!System.Web.Hosting.HostingEnvironment.Initialize+0x2b8
5f 071dbd8c 6db4f066     System_Web!System.Web.Hosting.HostingEnvironment.Initialize+0x16
60 071dbda8 6db5234a     clr!CallDescrWorkerInternal+0x34
61 071dbdfc 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
62 071dbe40 6dc1cbed     clr!DispatchCallDebuggerWrapper+0x59
63 071dbea0 6dc1dd92     clr!DispatchCall+0x7d
64 (Inline) --------     clr!RemotableMethodInfo::IsReturnGCRef+0xfffffff7
65 071dc048 6dc1ecaf     clr!CrossDomainChannel::MarshalAndCall_Wrapper+0x4c9
66 071dc054 6de314fe     clr!MarshalAndCall_Wrapper2+0xf
67 071dc09c 6de3158c     clr!MakeADCallDebuggerWrapper+0x5f
68 071dc124 6dc1eae6     clr!MakeCallWithAppDomainTransition+0x8f
69 071dd5cc 6dc1cccf     clr!CrossDomainChannel::MarshalAndCall+0x7a8
6a 071dd5f4 6dc1cdbb     clr!CrossDomainChannel::ExecuteCrossDomainCall+0x60
6b 071dd608 6dc1ce7e     clr!CrossDomainChannel::CheckCrossDomainCall+0xa8
6c 071dd728 6db4f566     clr!TransparentProxyStubWorker+0xbd
6d 071dd758 6a75b298     clr!TransparentProxyStub_CrossContext+0x14
6e 071dda24 6a759986     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment+0x18a4
6f 071dda54 6a7598f5     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors+0x26
70 071dda98 6a759843     System_Web_ni!System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment+0x6d
71 071ddab8 6a758ef5     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateObjectInternal+0x4f
72 071ddb0c 6adaaaab     System_Web_ni!System.Web.Hosting.ProcessHost.StartApplication+0x195
73 071ddb60 6db4f434     System_Web_ni+0x82aaab
74 071ddb88 6dc14143     clr!COMToCLRDispatchHelper+0x6b
75 (Inline) --------     clr!InvokeStub+0x3e
76 (Inline) --------     clr!COMToCLRInvokeTarget+0x5b
77 (Inline) --------     clr!COMToCLRWorkerBody+0x14f
78 071ddbe0 083ed085     clr!COMToCLRWorker+0x4d5
79 071ddc0c 67517444     CLRStub[StubLinkStub]@140a320d083ed085
7a 071ddc5c 6750bc76     webengine4!W3_MGD_APP_CONTEXT::CreateAppDomain+0xbb
7b 071ddc7c 6750bd3e     webengine4!W3_MGD_APP_CONTEXT::OnApplicationResolveModules+0xa9
7c 071ddc94 798071cb     webengine4!CMgdEngGlobalModule::OnGlobalApplicationResolveModules+0x5d
7d 071ddca8 797eb767     iiscore!VIRTUAL_MODULE::GlobalDoWork+0x15b
7e 071ddcd8 79808249     iiscore!W3_SERVER::GlobalNotify+0x115
7f 071ddd10 7980830f     iiscore!W3_APPLICATION::ResolveModules+0x22
80 071ddd10 797f0f8e     iiscore!W3_APPLICATION::SetupNotificationContext+0x86
81 071ddd58 797f0686     iiscore!W3_CONTEXT::SetupStateMachinePhase2+0x3e2
82 071dfddc 797f8607     iiscore!W3_CONTEXT::SetupStateMachine+0x2b8
83 071dfdf4 797f8b24     iiscore!W3_CONTEXT_BASE::StartNotificationLoop+0x72
84 071dfe0c 79836d9f     iiscore!W3_MAIN_CONTEXT::OnNewRequest+0xb4
85 (Inline) --------     w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x17
86 071dfe28 79837b8b     w3dt!UL_NATIVE_REQUEST::DoWork+0xa6
87 071dfe40 7983638f     w3dt!UL_RECEIVE_CONTEXT::DoWork+0x1b
88 071dfe5c 7982329b     w3dt!WP_CONTEXT::OnCompletion+0x2f
89 071dfe90 798231bb     w3tp!THREAD_POOL_DATA::ThreadPoolThread+0xa2
8a 071dfeb0 798220fc     w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x2b
8b 071dfecc 7558fa29     w3tp!THREAD_MANAGER::ThreadManagerThread+0x5c
8c 071dfedc 77497a7e     kernel32!BaseThreadInitThunk+0x19
8d 071dff38 77497a4e     ntdll!__RtlUserThreadStart+0x2f
8e 071dff48 00000000     ntdll!_RtlUserThreadStart+0x1b
0:005> !CLRStack
OS Thread Id: 0x5e80 (5)
Child SP       IP Call Site
071da2a0 774a46bc [HelperMethodFrame: 071da2a0] 
071da32c 21c6acea Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingScheduler.ScheduleToRunEveryTimeIntervalInMilliseconds(Int32, System.Action)
071da348 21c6ac0d Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingManager`1[[System.__Canon, mscorlib]]..ctor(System.__Canon, Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.IDiagnoisticsEventThrottlingScheduler, UInt32)
071da364 21c6aa42 Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da39c 21c6a865 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponent(System.Object, Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da3c8 21c6a90a Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents(System.Collections.IEnumerable, Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da3f0 21c6a815 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules)
071da400 21c68e63 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules, System.String)
071da438 21c6851f Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules)
071da44c 21c66445 Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.get_Active()
071da478 21c65297 Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor(System.String)
071da48c 21c64e5b Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor()
071da7a0 6db4f066 [DebuggerU2MCatchHandlerFrame: 071da7a0] 
071da560 6db4f066 [HelperMethodFrame_PROTECTOBJ: 071da560] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
071da83c 6ca85ce9 System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
071da868 2355b9f3 System.SecurityUtils.ConstructorInfoInvoke(System.Reflection.ConstructorInfo, System.Object[])
071da87c 2355b421 System.Diagnostics.TraceUtils.GetRuntimeObject(System.String, System.Type, System.String)
071da8bc 2355b2d8 System.Diagnostics.TypedElement.BaseGetRuntimeObject()
071da8d0 2355afc1 System.Diagnostics.ListenerElement.GetRuntimeObject()
071da8fc 2355a92e System.Diagnostics.ListenerElementsCollection.GetRuntimeObject()
071da930 2355a757 System.Diagnostics.TraceInternal.get_Listeners()
071dab78 6db4f066 [GCFrame: 071dab78] 
071dab5c 6db4f066 [GCFrame: 071dab5c] 
071dace0 6db4f066 [ProtectValueClassFrame: 071dace0] 
071dac30 6db4f066 [GCFrame: 071dac30] 
071dac14 6db4f066 [GCFrame: 071dac14] 
071dabf8 6db4f066 [GCFrame: 071dabf8] 
071daf08 6db4f066 [FuncEvalFrame: 071daf08] 
071daf54 21c6430e System.Diagnostics.DiagnosticSourceEventSource.BreakPointWithDebuggerFuncEval()
071daf64 21c64374 System.Diagnostics.DiagnosticSourceEventSource.OnEventCommand(System.Diagnostics.Tracing.EventCommandEventArgs)
071daf94 6d2542d2 System.Diagnostics.Tracing.EventSource.DoCommand(System.Diagnostics.Tracing.EventCommandEventArgs)
071dafdc 6ca6a986 System.Diagnostics.Tracing.EventSource.Initialize(System.Guid, System.String, System.String[])
071db024 6ca4d87b System.Diagnostics.Tracing.EventSource..ctor(System.Diagnostics.Tracing.EventSourceSettings, System.String[])
071db048 21c642e8 System.Diagnostics.DiagnosticSourceEventSource..ctor()
071db04c 21c64297 System.Diagnostics.DiagnosticSourceEventSource..cctor()
071db1cc 6db4f066 [GCFrame: 071db1cc] 
071db6ac 6db4f066 [HelperMethodFrame: 071db6ac] 
071db738 21c64207 System.Diagnostics.DiagnosticListener..ctor(System.String)
071db764 21c6412e Azure.Core.Pipeline.DiagnosticScopeFactory..ctor(System.String, System.String, Boolean)
071db77c 21c6408e Azure.Core.Pipeline.ClientDiagnostics..ctor(Azure.Core.ClientOptions)
071db794 21c63529 Azure.Identity.CredentialPipeline..ctor(Azure.Identity.TokenCredentialOptions)
071db7ac 21c62437 Azure.Identity.CredentialPipeline+c.b__19_0()
071db7bc 6ca7c49b System.Lazy`1[[System.__Canon, mscorlib]].CreateValue()
071db7e8 6ca7c10f System.Lazy`1[[System.__Canon, mscorlib]].LazyInitValue()
071db81c 6ca7c040 System.Lazy`1[[System.__Canon, mscorlib]].get_Value()
071db82c 21c61b83 Azure.Identity.CredentialPipeline.GetInstance(Azure.Identity.TokenCredentialOptions)
071db83c 21c621d6 Azure.Identity.DefaultAzureCredential..ctor(Azure.Identity.DefaultAzureCredentialOptions)
071db850 21c616cf Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder.LazyInitialize(System.String, System.Collections.Specialized.NameValueCollection)
071db890 21c605ac Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureInitialized()
071db8bc 21c60646 Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.GetValueInternal(System.String)
071db8ec 21c60fa0 Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.ProcessConfigurationSection(System.Configuration.ConfigurationSection)
071db930 21c7686d System.Configuration.Internal.InternalConfigHost.System.Configuration.Internal.IInternalConfigurationBuilderHost.ProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db93c 21c7689b System.Configuration.Internal.DelegatingConfigHost.ProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db948 21c76973 System.Configuration.BaseConfigurationRecord.CallHostProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db974 21b2f5b8 System.Configuration.RuntimeConfigurationRecord+RuntimeConfigurationFactory.CreateSectionImpl(System.Configuration.RuntimeConfigurationRecord, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071db9a0 21b85152 System.Configuration.RuntimeConfigurationRecord+RuntimeConfigurationFactory.CreateSectionWithRestrictedPermissions(System.Configuration.RuntimeConfigurationRecord, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071db9e4 21b2ec7a System.Configuration.RuntimeConfigurationRecord.CreateSection(Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071dba0c 21b2eb24 System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071dba58 21b341b8 System.Configuration.BaseConfigurationRecord.EvaluateOne(System.String[], System.Configuration.SectionInput, Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Object)
071dba90 21b33f1f System.Configuration.BaseConfigurationRecord.Evaluate(System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Object, Boolean, Boolean, System.Object ByRef, System.Object ByRef)
071dbb54 21b2df56 System.Configuration.BaseConfigurationRecord.GetSectionRecursive(System.String, Boolean, Boolean, Boolean, Boolean, System.Object ByRef, System.Object ByRef)
071dbbd4 21b2db23 System.Configuration.BaseConfigurationRecord.GetSection(System.String)
071dbbe4 21bafe86 System.Web.Util.AppSettings.GetAppSettingsSection()
071dbbf4 21c71339 System.Web.Util.AppSettings.EnsureSettingsLoaded()
071dbcfc 21c720a7 System.Web.Compilation.BuildManager.RestorePortableCompilationOutputSnapshot()
071dbd10 21c722b4 System.Web.Compilation.BuildManager.ExecutePreAppStart()
071dbd40 085b3098 System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel, System.Exception)
071dbd8c 085b2026 System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
071dbf18 6db4f066 [GCFrame: 071dbf18] 
071dbefc 6db4f066 [GCFrame: 071dbefc] 
071dc0bc 6db4f066 [ContextTransitionFrame: 071dc0bc] 
071dc218 6db4f066 [GCFrame: 071dc218] 
071dc1fc 6db4f066 [GCFrame: 071dc1fc] 
071dd4d4 6db4f066 [GCSafeCollectionFrame: 071dd4d4] 
071dd4e4 6db4f066 [GCSafeCollectionFrame: 071dd4e4] 
071dd4f4 6db4f066 [GCSafeCollectionFrame: 071dd4f4] 
071dd504 6db4f066 [GCSafeCollectionFrame: 071dd504] 
071dd514 6db4f066 [GCSafeCollectionFrame: 071dd514] 
071dd524 6db4f066 [GCSafeCollectionFrame: 071dd524] 
071dd534 6db4f066 [GCSafeCollectionFrame: 071dd534] 
071dd704 6db4f066 [TPMethodFrame: 071dd704] System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
071dd770 6a75b298 System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071dda34 6a759986 System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071dda64 6a7598f5 System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071ddaa8 6a759843 System.Web.Hosting.ApplicationManager.CreateObjectInternal(System.String, System.Type, System.Web.Hosting.IApplicationHost, Boolean, System.Web.Hosting.HostingEnvironmentParameters)
071ddad0 6a758ef5 System.Web.Hosting.ProcessHost.StartApplication(System.String, System.String, System.Object ByRef)
071ddb1c 6adaaaab DomainNeutralILStubClass.IL_STUB_COMtoCLR(IntPtr, IntPtr, IntPtr)
071ddbf4 6db4f434 [ComMethodFrame: 071ddbf4]

@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Configuration untriaged New issue has not been triaged by the area owner labels Jul 15, 2021
@ghost
Copy link

ghost commented Jul 15, 2021

Tagging subscribers to this area: @safern
See info in area-owners.md if you want to be subscribed.

Issue Details

I represent the Application Insights .NET SDK.
Our customers are reporting an issue with our TraceListener.

image

Description

Per our investigation, we're seeing deadlocks involving clr native code and System.Diagnostics.TraceInternal and need help investigating.

Related issues:

Other information

We had a customer kindly share a repro here:

We investigated using WinDbg and found evidence of deadlocks.

DEADLOCK DETECTED
Thread 49 holds CriticalSection 23016cd8
...and is waiting for the lock on SyncBlock 07bc55ac OBJ:0e7ea3f4[System.Collections.Generic.List1[[System.WeakReference, mscorlib]]] Thread 5 holds the lock on SyncBlock 07bc55ac OBJ:0e7ea3f4[System.Collections.Generic.List1[[System.WeakReference, mscorlib]]]
...and is waiting on CriticalSection 23016cd8
Thread 49 is waiting at System.Diagnostics.Tracing.EventListener.AddEventSource(System.Diagnostics.Tracing.EventSource)(+50 IL,+0x42 Native)
Thread 5 is waiting at clr!CrstBase::SpinEnter+0xd6

Thread 49 => clr code => Entered critsec => Executing code => Managed code => Waiting to enter a managed lock
https://github.com/microsoft/ApplicationInsights-dotnet/blob/317786702ee9c356f7e452b6739fcdcceb59f455/BASE/src/Microsoft.ApplicationInsights/Extensibility/Implementation/Tracing/CoreEventSource.cs#L19

0:049> kL
# ChildEBP RetAddr      
00 23bbe080 76f3b143     ntdll!NtWaitForMultipleObjects+0xc
01 23bbe080 6dc54541     KERNELBASE!WaitForMultipleObjectsEx+0x103
02 23bbe0d0 6dc54190     clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x3c
03 (Inline) --------     clr!Thread::DoAppropriateAptStateWait+0x35
04 23bbe154 6dc542be     clr!Thread::DoAppropriateWaitWorker+0x1eb
05 23bbe1c0 6dc54407     clr!Thread::DoAppropriateWait+0x64
06 (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException+0xfffffffe
07 23bbe20c 6db633db     clr!CLREventBase::WaitEx+0x121
08 23bbe220 6dc3f29b     clr!CLREventBase::Wait+0x1a
09 23bbe2ac 6dc3f3cc     clr!AwareLock::EnterEpilogHelper+0xa8
0a (Inline) --------     clr!ThreadDebugBlockingInfo::PopBlockingItem+0xfffffff9
0b (Inline) --------     clr!DebugBlockingItemHolder::{dtor}+0xfffffff9
0c 23bbe2f4 6dc3f1a5     clr!AwareLock::EnterEpilog+0x48
0d 23bbe3b8 6dc71152     clr!AwareLock::Enter+0x4a
0e (Inline) --------     clr!SyncBlock::EnterMonitor+0x7
0f (Inline) --------     clr!ObjHeader::EnterObjMonitor+0x12
10 (Inline) --------     clr!Object::EnterObjMonitor+0x18
11 23bbe3b8 6ca6b012     clr!JITutil_MonReliableEnter+0xb5
12 23bbe3f0 6ca6a753     mscorlib_ni!System.Diagnostics.Tracing.EventListener.AddEventSource+0x42
13 23bbe424 6ca4d87b     mscorlib_ni!System.Diagnostics.Tracing.EventSource.Initialize+0x11f
14 23bbe458 21c68317     mscorlib_ni!System.Diagnostics.Tracing.EventSource..ctor+0x4f
15 23bbe46c 21c682b7     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.CoreEventSource..ctor+0x37
16 23bbe478 6db4f066     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.CoreEventSource..cctor+0x1f
17 23bbe484 6db5234a     clr!CallDescrWorkerInternal+0x34
18 23bbe4d8 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
19 23bbe51c 6db64392     clr!DispatchCallDebuggerWrapper+0x59
1a 23bbe568 6dc54fde     clr!DispatchCallSimple+0x97
1b 23bbe5d4 6dc550ab     clr!MethodTable::RunClassInitEx+0xca
1c 23bbeab8 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x45d
1d (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
1e 23bbeb40 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
1f 23bbeb74 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
WARNING: Frame IP not in any known module. Following frames may be wrong.
20 23bbeb74 21c67ff7     0x83e34b1
21 23bbebb4 21c67e3c     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.DequeueAndSend+0x5f
22 23bbebe4 6cabd4bb     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Runner+0x54
23 23bbebf0 6cabb731     mscorlib_ni!System.Threading.Tasks.Task.InnerInvoke+0x4b
24 23bbec14 6cabb6fc     mscorlib_ni!System.Threading.Tasks.Task.Execute+0x31
25 23bbec7c 6ca58604     mscorlib_ni!System.Threading.Tasks.Task.ExecutionContextCallback+0x1c
26 23bbec7c 6ca58537     mscorlib_ni!System.Threading.ExecutionContext.RunInternal+0xc4
27 23bbec90 6cabb4b2     mscorlib_ni!System.Threading.ExecutionContext.Run+0x17
28 23bbecfc 6cabb357     mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0xe2
29 23bbed0c 6cad6d0a     mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry+0xb7
2a 23bbed1c 6ca32dd6     mscorlib_ni!System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork+0x1e
2b 23bbed1c 6ca58604     mscorlib_ni!System.Threading.ThreadHelper.ThreadStart_Context+0x76
2c 23bbed80 6ca58537     mscorlib_ni!System.Threading.ExecutionContext.RunInternal+0xc4
2d 23bbed94 6ca584f4     mscorlib_ni!System.Threading.ExecutionContext.Run+0x17
2e 23bbedb0 6ca32d0e     mscorlib_ni!System.Threading.ExecutionContext.Run+0x44
2f 23bbedc8 6db4f066     mscorlib_ni!System.Threading.ThreadHelper.ThreadStart+0x52
30 23bbedd4 6db5234a     clr!CallDescrWorkerInternal+0x34
31 23bbee28 6db5862b     clr!CallDescrWorkerWithHandler+0x6b
32 23bbee9c 6dc69727     clr!MethodDescCallSite::CallTargetWorker+0x16a
33 (Inline) --------     clr!MethodDescCallSite::Call+0x12
34 23bbf00c 6dcfe376     clr!ThreadNative::KickOffThread_Worker+0x131
35 23bbf020 6dcfe401     clr!ManagedThreadBase_DispatchInner+0x71
36 (Inline) --------     clr!ManagedThreadBase_DispatchMiddle::__l16::Cleanup::{dtor}+0xfffffffc
37 23bbf0c4 6dcfe2f2     clr!ManagedThreadBase_DispatchMiddle+0x7e
38 23bbf118 6dd01655     clr!ManagedThreadBase_DispatchOuter+0x99
39 23bbf124 6dd015e1     clr!ManagedThreadBase_DispatchInCorrectAD+0x15
3a 23bbf1f0 6dd01680     clr!Thread::DoADCallBack+0x328
3b 23bbf214 6dcfe401     clr!ManagedThreadBase_DispatchInner+0x4e
3c (Inline) --------     clr!ManagedThreadBase_DispatchMiddle::__l16::Cleanup::{dtor}+0xfffffffc
3d 23bbf2b8 6dcfe2f2     clr!ManagedThreadBase_DispatchMiddle+0x7e
3e 23bbf30c 6dcfe4e1     clr!ManagedThreadBase_DispatchOuter+0x99
3f 23bbf330 6dc695d8     clr!ManagedThreadBase_FullTransitionWithAD+0x2f
40 23bbf3b4 6dc14c87     clr!ThreadNative::KickOffThread+0x260
41 23bbf9d8 7558fa29     clr!Thread::intermediateThreadProc+0x58
42 23bbf9e8 77497a7e     kernel32!BaseThreadInitThunk+0x19
43 23bbfa44 77497a4e     ntdll!__RtlUserThreadStart+0x2f
44 23bbfa54 00000000     ntdll!_RtlUserThreadStart+0x1b

Thread 5 => clr code => managed code => Entered lock => executing code => clr code => Waiting to enter critsec
https://github.com/microsoft/ApplicationInsights-dotnet/blob/317786702ee9c356f7e452b6739fcdcceb59f455/BASE/src/Microsoft.ApplicationInsights/Extensibility/Implementation/Tracing/DiagnosticsModule/DiagnoisticsEventThrottlingScheduler.cs#L48
https://github.com/microsoft/ApplicationInsights-dotnet/blob/317786702ee9c356f7e452b6739fcdcceb59f455/BASE/src/Microsoft.ApplicationInsights/Extensibility/Implementation/Tracing/CoreEventSource.cs#L111

0:005> kL
# ChildEBP RetAddr      
00 071d9c58 7748f909     ntdll!NtWaitForAlertByThreadId+0xc
01 071d9c58 7748f65d     ntdll!RtlpWaitOnAddressWithTimeout+0x64
02 (Inline) --------     ntdll!RtlpWaitOnAddress+0xbb
03 071d9cf8 774700ca     ntdll!RtlpWaitOnCriticalSection+0x18d
04 071d9d30 7746ff19     ntdll!RtlpEnterCriticalSectionContended+0x1aa
05 071d9d3c 6dc5b360     ntdll!RtlEnterCriticalSection+0x49
06 (Inline) --------     clr!UnsafeEnterCriticalSection+0x7
07 071d9d74 6dc5b2bb     clr!CrstBase::SpinEnter+0xd6
08 071d9d74 6db64b9c     clr!CrstBase::Enter+0x136
09 (Inline) --------     clr!Volatile<Thread *>::{ctor}+0xfffffffb
0a (Inline) --------     clr!VolatilePtr<Thread,Thread *>::{ctor}+0xfffffffb
0b (Inline) --------     clr!DeadlockAwareLock::EndEnterLock
0c (Inline) --------     clr!ListLockEntry::FinishDeadlockAwareEnter+0x1e
0d 071d9da4 6dc54e02     clr!ListLockEntry::LockHolder::DeadlockAwareAcquire+0x3e
0e 071da284 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x3e8
0f (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
10 071da30c 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
11 071da33c 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
WARNING: Frame IP not in any known module. Following frames may be wrong.
12 071da33c 21c6ac0d     0x83e34b1
13 071da354 21c6aa42     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingManager<System.__Canon>..ctor+0x6d
14 071da394 21c6a865     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule.Initialize+0xba
15 071da3c0 21c6a90a     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponent+0x35
16 071da3e8 21c6a815     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents+0x42
17 071da3f8 21c68e63     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents+0x4d
18 071da428 21c6851f     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize+0x22b
19 071da440 21c66445     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize+0x27
1a 071da470 21c65297     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.get_Active+0x95
1b 071da484 21c64e5b     Microsoft_ApplicationInsights_TraceListener!Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor+0x27
1c 071da494 6db4f066     Microsoft_ApplicationInsights_TraceListener!Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor+0xb
1d 071da494 6db5234a     clr!CallDescrWorkerInternal+0x34
1e 071da4e8 6db677f1     clr!CallDescrWorkerWithHandler+0x6b
1f 071da528 6db678f6     clr!CallDescrWorkerReflectionWrapper+0x55
20 071da82c 6ca85ce9     clr!RuntimeMethodHandle::InvokeMethod+0x838
21 071da854 2355b9f3     mscorlib_ni!System.Reflection.RuntimeConstructorInfo.Invoke+0x149
22 071da874 2355b421     System!System.SecurityUtils.ConstructorInfoInvoke+0x5b
23 071da8b0 2355b2d8     System!System.Diagnostics.TraceUtils.GetRuntimeObject+0xd9
24 071da8c8 2355afc1     System!System.Diagnostics.TypedElement.BaseGetRuntimeObject+0x30
25 071da8f4 2355a92e     System!System.Diagnostics.ListenerElement.GetRuntimeObject+0x1b1
26 071da928 2355a757     System!System.Diagnostics.ListenerElementsCollection.GetRuntimeObject+0xb6
27 071da958 6db4f066     System!System.Diagnostics.TraceInternal.get_Listeners+0x9f
28 071da964 6db5234a     clr!CallDescrWorkerInternal+0x34
29 071da9b8 6db5862b     clr!CallDescrWorkerWithHandler+0x6b
2a 071daa24 6e0b669c     clr!MethodDescCallSite::CallTargetWorker+0x16a
2b (Inline) --------     clr!MethodDescCallSite::CallWithValueTypes_RetArgSlot+0x6
2c 071daa70 6e0b6ba5     clr!FuncEvalWrapper+0x2e
2d 071dabc4 6e0b6f09     clr!DoNormalFuncEval+0x2b6
2e 071dad24 6e0b7c36     clr!GCProtectArgsAndDoNormalFuncEval+0x201
2f 071daed0 6e0b7dc1     clr!FuncEvalHijackRealWorker+0x796
30 071daf48 6db41186     clr!FuncEvalHijackWorker+0x14d
31 071daf5c 21c64374     clr!FuncEvalHijack+0x6
32 071daf8c 6d2542d2     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource.OnEventCommand+0x24
33 071dafd4 6ca6a986     mscorlib_ni!System.Diagnostics.Tracing.EventSource.DoCommand+0x21e
34 071db008 6ca4d87b     mscorlib_ni!System.Diagnostics.Tracing.EventSource.Initialize+0x352
35 071db03c 21c642e8     mscorlib_ni!System.Diagnostics.Tracing.EventSource..ctor+0x4f
36 071db050 21c64297     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource..ctor+0x28
37 071db050 6db4f066     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource..cctor+0x1f
38 071db05c 6db5234a     clr!CallDescrWorkerInternal+0x34
39 071db0b0 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
3a 071db0f4 6db64392     clr!DispatchCallDebuggerWrapper+0x59
3b 071db140 6dc54fde     clr!DispatchCallSimple+0x97
3c 071db1ac 6dc550ab     clr!MethodTable::RunClassInitEx+0xca
3d 071db690 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x45d
3e (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
3f 071db718 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
40 071db75c 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
41 071db75c 21c6412e     0x83e34b1
42 071db78c 21c6408e     Azure_Identity!+0x36
43 071db78c 21c63529     Azure_Identity!Azure.Core.Pipeline.ClientDiagnostics..ctor+0x4e
44 071db7a4 21c62437     Azure_Identity!+0x71
45 071db7b4 6ca7c49b     Azure_Identity!+0x2f
46 071db7e0 6ca7c10f     mscorlib_ni!System.Lazy`1.CreateValue+0xbb
47 071db814 6ca7c040     mscorlib_ni!System.Lazy`1.LazyInitValue+0x9f
48 071db824 21c61b83     mscorlib_ni!System.Lazy`1.get_Value+0x50
49 071db834 21c621d6     Azure_Identity!+0x43
4a 071db848 21c616cf     Azure_Identity!Azure.Identity.DefaultAzureCredential..ctor+0x26
4b 071db884 21c605ac     Microsoft_Configuration_ConfigurationBuilders_Azure!Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder.LazyInitialize+0x1b7
4c 071db8b4 21c60646     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureInitialized+0x5c
4d 071db8e4 21c60fa0     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.GetValueInternal+0x4e
4e 071db928 21c7686d     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.ProcessConfigurationSection+0xd0
4f 071db930 21c7689b     System_Configuration!System.Configuration.Internal.InternalConfigHost.System.Configuration.Internal.IInternalConfigurationBuilderHost.ProcessConfigurationSection+0x15
50 071db93c 21c76973     System_Configuration!System.Configuration.Internal.DelegatingConfigHost.ProcessConfigurationSection+0x13
51 071db968 21b2f5b8     System_Configuration!System.Configuration.BaseConfigurationRecord.CallHostProcessConfigurationSection+0x33
52 071db984 21b85152     System_Configuration!System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionImpl+0xc8
53 071db9c8 21b2ec7a     System_Configuration!System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionWithRestrictedPermissions+0x62
54 071db9f0 21b2eb24     System_Configuration!System.Configuration.RuntimeConfigurationRecord.CreateSection+0x62
55 071dba3c 21b341b8     System_Configuration!System.Configuration.BaseConfigurationRecord.CallCreateSection+0x6c
56 071dba74 21b33f1f     System_Configuration!System.Configuration.BaseConfigurationRecord.EvaluateOne+0x58
57 071dbb34 21b2df56     System_Configuration!System.Configuration.BaseConfigurationRecord.Evaluate+0x43f
58 071dbbb4 21b2db23     System_Configuration!System.Configuration.BaseConfigurationRecord.GetSectionRecursive+0x416
59 071dbbdc 21bafe86     System_Configuration!System.Configuration.BaseConfigurationRecord.GetSection+0x2b
5a 071dbbec 21c71339     System_Web!System.Web.Util.AppSettings.GetAppSettingsSection+0x56
5b 071dbcf4 21c720a7     System_Web!System.Web.Util.AppSettings.EnsureSettingsLoaded+0x89
5c 071dbd08 21c722b4     System_Web!System.Web.Compilation.BuildManager.RestorePortableCompilationOutputSnapshot+0x1f
5d 071dbd38 085b3098     System_Web!System.Web.Compilation.BuildManager.ExecutePreAppStart+0x14
5e 071dbd70 085b2026     System_Web!System.Web.Hosting.HostingEnvironment.Initialize+0x2b8
5f 071dbd8c 6db4f066     System_Web!System.Web.Hosting.HostingEnvironment.Initialize+0x16
60 071dbda8 6db5234a     clr!CallDescrWorkerInternal+0x34
61 071dbdfc 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
62 071dbe40 6dc1cbed     clr!DispatchCallDebuggerWrapper+0x59
63 071dbea0 6dc1dd92     clr!DispatchCall+0x7d
64 (Inline) --------     clr!RemotableMethodInfo::IsReturnGCRef+0xfffffff7
65 071dc048 6dc1ecaf     clr!CrossDomainChannel::MarshalAndCall_Wrapper+0x4c9
66 071dc054 6de314fe     clr!MarshalAndCall_Wrapper2+0xf
67 071dc09c 6de3158c     clr!MakeADCallDebuggerWrapper+0x5f
68 071dc124 6dc1eae6     clr!MakeCallWithAppDomainTransition+0x8f
69 071dd5cc 6dc1cccf     clr!CrossDomainChannel::MarshalAndCall+0x7a8
6a 071dd5f4 6dc1cdbb     clr!CrossDomainChannel::ExecuteCrossDomainCall+0x60
6b 071dd608 6dc1ce7e     clr!CrossDomainChannel::CheckCrossDomainCall+0xa8
6c 071dd728 6db4f566     clr!TransparentProxyStubWorker+0xbd
6d 071dd758 6a75b298     clr!TransparentProxyStub_CrossContext+0x14
6e 071dda24 6a759986     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment+0x18a4
6f 071dda54 6a7598f5     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors+0x26
70 071dda98 6a759843     System_Web_ni!System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment+0x6d
71 071ddab8 6a758ef5     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateObjectInternal+0x4f
72 071ddb0c 6adaaaab     System_Web_ni!System.Web.Hosting.ProcessHost.StartApplication+0x195
73 071ddb60 6db4f434     System_Web_ni+0x82aaab
74 071ddb88 6dc14143     clr!COMToCLRDispatchHelper+0x6b
75 (Inline) --------     clr!InvokeStub+0x3e
76 (Inline) --------     clr!COMToCLRInvokeTarget+0x5b
77 (Inline) --------     clr!COMToCLRWorkerBody+0x14f
78 071ddbe0 083ed085     clr!COMToCLRWorker+0x4d5
79 071ddc0c 67517444     CLRStub[StubLinkStub]@140a320d083ed085
7a 071ddc5c 6750bc76     webengine4!W3_MGD_APP_CONTEXT::CreateAppDomain+0xbb
7b 071ddc7c 6750bd3e     webengine4!W3_MGD_APP_CONTEXT::OnApplicationResolveModules+0xa9
7c 071ddc94 798071cb     webengine4!CMgdEngGlobalModule::OnGlobalApplicationResolveModules+0x5d
7d 071ddca8 797eb767     iiscore!VIRTUAL_MODULE::GlobalDoWork+0x15b
7e 071ddcd8 79808249     iiscore!W3_SERVER::GlobalNotify+0x115
7f 071ddd10 7980830f     iiscore!W3_APPLICATION::ResolveModules+0x22
80 071ddd10 797f0f8e     iiscore!W3_APPLICATION::SetupNotificationContext+0x86
81 071ddd58 797f0686     iiscore!W3_CONTEXT::SetupStateMachinePhase2+0x3e2
82 071dfddc 797f8607     iiscore!W3_CONTEXT::SetupStateMachine+0x2b8
83 071dfdf4 797f8b24     iiscore!W3_CONTEXT_BASE::StartNotificationLoop+0x72
84 071dfe0c 79836d9f     iiscore!W3_MAIN_CONTEXT::OnNewRequest+0xb4
85 (Inline) --------     w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x17
86 071dfe28 79837b8b     w3dt!UL_NATIVE_REQUEST::DoWork+0xa6
87 071dfe40 7983638f     w3dt!UL_RECEIVE_CONTEXT::DoWork+0x1b
88 071dfe5c 7982329b     w3dt!WP_CONTEXT::OnCompletion+0x2f
89 071dfe90 798231bb     w3tp!THREAD_POOL_DATA::ThreadPoolThread+0xa2
8a 071dfeb0 798220fc     w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x2b
8b 071dfecc 7558fa29     w3tp!THREAD_MANAGER::ThreadManagerThread+0x5c
8c 071dfedc 77497a7e     kernel32!BaseThreadInitThunk+0x19
8d 071dff38 77497a4e     ntdll!__RtlUserThreadStart+0x2f
8e 071dff48 00000000     ntdll!_RtlUserThreadStart+0x1b
0:005> !CLRStack
OS Thread Id: 0x5e80 (5)
Child SP       IP Call Site
071da2a0 774a46bc [HelperMethodFrame: 071da2a0] 
071da32c 21c6acea Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingScheduler.ScheduleToRunEveryTimeIntervalInMilliseconds(Int32, System.Action)
071da348 21c6ac0d Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingManager`1[[System.__Canon, mscorlib]]..ctor(System.__Canon, Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.IDiagnoisticsEventThrottlingScheduler, UInt32)
071da364 21c6aa42 Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da39c 21c6a865 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponent(System.Object, Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da3c8 21c6a90a Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents(System.Collections.IEnumerable, Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da3f0 21c6a815 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules)
071da400 21c68e63 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules, System.String)
071da438 21c6851f Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules)
071da44c 21c66445 Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.get_Active()
071da478 21c65297 Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor(System.String)
071da48c 21c64e5b Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor()
071da7a0 6db4f066 [DebuggerU2MCatchHandlerFrame: 071da7a0] 
071da560 6db4f066 [HelperMethodFrame_PROTECTOBJ: 071da560] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
071da83c 6ca85ce9 System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
071da868 2355b9f3 System.SecurityUtils.ConstructorInfoInvoke(System.Reflection.ConstructorInfo, System.Object[])
071da87c 2355b421 System.Diagnostics.TraceUtils.GetRuntimeObject(System.String, System.Type, System.String)
071da8bc 2355b2d8 System.Diagnostics.TypedElement.BaseGetRuntimeObject()
071da8d0 2355afc1 System.Diagnostics.ListenerElement.GetRuntimeObject()
071da8fc 2355a92e System.Diagnostics.ListenerElementsCollection.GetRuntimeObject()
071da930 2355a757 System.Diagnostics.TraceInternal.get_Listeners()
071dab78 6db4f066 [GCFrame: 071dab78] 
071dab5c 6db4f066 [GCFrame: 071dab5c] 
071dace0 6db4f066 [ProtectValueClassFrame: 071dace0] 
071dac30 6db4f066 [GCFrame: 071dac30] 
071dac14 6db4f066 [GCFrame: 071dac14] 
071dabf8 6db4f066 [GCFrame: 071dabf8] 
071daf08 6db4f066 [FuncEvalFrame: 071daf08] 
071daf54 21c6430e System.Diagnostics.DiagnosticSourceEventSource.BreakPointWithDebuggerFuncEval()
071daf64 21c64374 System.Diagnostics.DiagnosticSourceEventSource.OnEventCommand(System.Diagnostics.Tracing.EventCommandEventArgs)
071daf94 6d2542d2 System.Diagnostics.Tracing.EventSource.DoCommand(System.Diagnostics.Tracing.EventCommandEventArgs)
071dafdc 6ca6a986 System.Diagnostics.Tracing.EventSource.Initialize(System.Guid, System.String, System.String[])
071db024 6ca4d87b System.Diagnostics.Tracing.EventSource..ctor(System.Diagnostics.Tracing.EventSourceSettings, System.String[])
071db048 21c642e8 System.Diagnostics.DiagnosticSourceEventSource..ctor()
071db04c 21c64297 System.Diagnostics.DiagnosticSourceEventSource..cctor()
071db1cc 6db4f066 [GCFrame: 071db1cc] 
071db6ac 6db4f066 [HelperMethodFrame: 071db6ac] 
071db738 21c64207 System.Diagnostics.DiagnosticListener..ctor(System.String)
071db764 21c6412e Azure.Core.Pipeline.DiagnosticScopeFactory..ctor(System.String, System.String, Boolean)
071db77c 21c6408e Azure.Core.Pipeline.ClientDiagnostics..ctor(Azure.Core.ClientOptions)
071db794 21c63529 Azure.Identity.CredentialPipeline..ctor(Azure.Identity.TokenCredentialOptions)
071db7ac 21c62437 Azure.Identity.CredentialPipeline+c.b__19_0()
071db7bc 6ca7c49b System.Lazy`1[[System.__Canon, mscorlib]].CreateValue()
071db7e8 6ca7c10f System.Lazy`1[[System.__Canon, mscorlib]].LazyInitValue()
071db81c 6ca7c040 System.Lazy`1[[System.__Canon, mscorlib]].get_Value()
071db82c 21c61b83 Azure.Identity.CredentialPipeline.GetInstance(Azure.Identity.TokenCredentialOptions)
071db83c 21c621d6 Azure.Identity.DefaultAzureCredential..ctor(Azure.Identity.DefaultAzureCredentialOptions)
071db850 21c616cf Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder.LazyInitialize(System.String, System.Collections.Specialized.NameValueCollection)
071db890 21c605ac Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureInitialized()
071db8bc 21c60646 Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.GetValueInternal(System.String)
071db8ec 21c60fa0 Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.ProcessConfigurationSection(System.Configuration.ConfigurationSection)
071db930 21c7686d System.Configuration.Internal.InternalConfigHost.System.Configuration.Internal.IInternalConfigurationBuilderHost.ProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db93c 21c7689b System.Configuration.Internal.DelegatingConfigHost.ProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db948 21c76973 System.Configuration.BaseConfigurationRecord.CallHostProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db974 21b2f5b8 System.Configuration.RuntimeConfigurationRecord+RuntimeConfigurationFactory.CreateSectionImpl(System.Configuration.RuntimeConfigurationRecord, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071db9a0 21b85152 System.Configuration.RuntimeConfigurationRecord+RuntimeConfigurationFactory.CreateSectionWithRestrictedPermissions(System.Configuration.RuntimeConfigurationRecord, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071db9e4 21b2ec7a System.Configuration.RuntimeConfigurationRecord.CreateSection(Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071dba0c 21b2eb24 System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071dba58 21b341b8 System.Configuration.BaseConfigurationRecord.EvaluateOne(System.String[], System.Configuration.SectionInput, Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Object)
071dba90 21b33f1f System.Configuration.BaseConfigurationRecord.Evaluate(System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Object, Boolean, Boolean, System.Object ByRef, System.Object ByRef)
071dbb54 21b2df56 System.Configuration.BaseConfigurationRecord.GetSectionRecursive(System.String, Boolean, Boolean, Boolean, Boolean, System.Object ByRef, System.Object ByRef)
071dbbd4 21b2db23 System.Configuration.BaseConfigurationRecord.GetSection(System.String)
071dbbe4 21bafe86 System.Web.Util.AppSettings.GetAppSettingsSection()
071dbbf4 21c71339 System.Web.Util.AppSettings.EnsureSettingsLoaded()
071dbcfc 21c720a7 System.Web.Compilation.BuildManager.RestorePortableCompilationOutputSnapshot()
071dbd10 21c722b4 System.Web.Compilation.BuildManager.ExecutePreAppStart()
071dbd40 085b3098 System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel, System.Exception)
071dbd8c 085b2026 System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
071dbf18 6db4f066 [GCFrame: 071dbf18] 
071dbefc 6db4f066 [GCFrame: 071dbefc] 
071dc0bc 6db4f066 [ContextTransitionFrame: 071dc0bc] 
071dc218 6db4f066 [GCFrame: 071dc218] 
071dc1fc 6db4f066 [GCFrame: 071dc1fc] 
071dd4d4 6db4f066 [GCSafeCollectionFrame: 071dd4d4] 
071dd4e4 6db4f066 [GCSafeCollectionFrame: 071dd4e4] 
071dd4f4 6db4f066 [GCSafeCollectionFrame: 071dd4f4] 
071dd504 6db4f066 [GCSafeCollectionFrame: 071dd504] 
071dd514 6db4f066 [GCSafeCollectionFrame: 071dd514] 
071dd524 6db4f066 [GCSafeCollectionFrame: 071dd524] 
071dd534 6db4f066 [GCSafeCollectionFrame: 071dd534] 
071dd704 6db4f066 [TPMethodFrame: 071dd704] System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
071dd770 6a75b298 System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071dda34 6a759986 System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071dda64 6a7598f5 System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071ddaa8 6a759843 System.Web.Hosting.ApplicationManager.CreateObjectInternal(System.String, System.Type, System.Web.Hosting.IApplicationHost, Boolean, System.Web.Hosting.HostingEnvironmentParameters)
071ddad0 6a758ef5 System.Web.Hosting.ProcessHost.StartApplication(System.String, System.String, System.Object ByRef)
071ddb1c 6adaaaab DomainNeutralILStubClass.IL_STUB_COMtoCLR(IntPtr, IntPtr, IntPtr)
071ddbf4 6db4f434 [ComMethodFrame: 071ddbf4]

Author: TimothyMothra
Assignees: -
Labels:

area-System.Configuration, untriaged

Milestone: -

@TimothyMothra
Copy link
Author

@rajkumar-rangaraj @cijothomas @reyang For reference

@ghost
Copy link

ghost commented Jul 15, 2021

Tagging subscribers to this area: @tarekgh, @tommcdon, @pjanotti
See info in area-owners.md if you want to be subscribed.

Issue Details

I represent the Application Insights .NET SDK.
Our customers are reporting an issue with our TraceListener.

image

Description

Per our investigation, we're seeing deadlocks involving clr native code and System.Diagnostics.TraceInternal and need help investigating.

Related issues:

Other information

We had a customer kindly share a repro here:

We investigated using WinDbg and found evidence of deadlocks.

DEADLOCK DETECTED
Thread 49 holds CriticalSection 23016cd8
...and is waiting for the lock on SyncBlock 07bc55ac OBJ:0e7ea3f4[System.Collections.Generic.List1[[System.WeakReference, mscorlib]]] Thread 5 holds the lock on SyncBlock 07bc55ac OBJ:0e7ea3f4[System.Collections.Generic.List1[[System.WeakReference, mscorlib]]]
...and is waiting on CriticalSection 23016cd8
Thread 49 is waiting at System.Diagnostics.Tracing.EventListener.AddEventSource(System.Diagnostics.Tracing.EventSource)(+50 IL,+0x42 Native)
Thread 5 is waiting at clr!CrstBase::SpinEnter+0xd6

Thread 49 => clr code => Entered critsec => Executing code => Managed code => Waiting to enter a managed lock
https://github.com/microsoft/ApplicationInsights-dotnet/blob/317786702ee9c356f7e452b6739fcdcceb59f455/BASE/src/Microsoft.ApplicationInsights/Extensibility/Implementation/Tracing/CoreEventSource.cs#L19

0:049> kL
# ChildEBP RetAddr      
00 23bbe080 76f3b143     ntdll!NtWaitForMultipleObjects+0xc
01 23bbe080 6dc54541     KERNELBASE!WaitForMultipleObjectsEx+0x103
02 23bbe0d0 6dc54190     clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x3c
03 (Inline) --------     clr!Thread::DoAppropriateAptStateWait+0x35
04 23bbe154 6dc542be     clr!Thread::DoAppropriateWaitWorker+0x1eb
05 23bbe1c0 6dc54407     clr!Thread::DoAppropriateWait+0x64
06 (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException+0xfffffffe
07 23bbe20c 6db633db     clr!CLREventBase::WaitEx+0x121
08 23bbe220 6dc3f29b     clr!CLREventBase::Wait+0x1a
09 23bbe2ac 6dc3f3cc     clr!AwareLock::EnterEpilogHelper+0xa8
0a (Inline) --------     clr!ThreadDebugBlockingInfo::PopBlockingItem+0xfffffff9
0b (Inline) --------     clr!DebugBlockingItemHolder::{dtor}+0xfffffff9
0c 23bbe2f4 6dc3f1a5     clr!AwareLock::EnterEpilog+0x48
0d 23bbe3b8 6dc71152     clr!AwareLock::Enter+0x4a
0e (Inline) --------     clr!SyncBlock::EnterMonitor+0x7
0f (Inline) --------     clr!ObjHeader::EnterObjMonitor+0x12
10 (Inline) --------     clr!Object::EnterObjMonitor+0x18
11 23bbe3b8 6ca6b012     clr!JITutil_MonReliableEnter+0xb5
12 23bbe3f0 6ca6a753     mscorlib_ni!System.Diagnostics.Tracing.EventListener.AddEventSource+0x42
13 23bbe424 6ca4d87b     mscorlib_ni!System.Diagnostics.Tracing.EventSource.Initialize+0x11f
14 23bbe458 21c68317     mscorlib_ni!System.Diagnostics.Tracing.EventSource..ctor+0x4f
15 23bbe46c 21c682b7     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.CoreEventSource..ctor+0x37
16 23bbe478 6db4f066     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.CoreEventSource..cctor+0x1f
17 23bbe484 6db5234a     clr!CallDescrWorkerInternal+0x34
18 23bbe4d8 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
19 23bbe51c 6db64392     clr!DispatchCallDebuggerWrapper+0x59
1a 23bbe568 6dc54fde     clr!DispatchCallSimple+0x97
1b 23bbe5d4 6dc550ab     clr!MethodTable::RunClassInitEx+0xca
1c 23bbeab8 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x45d
1d (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
1e 23bbeb40 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
1f 23bbeb74 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
WARNING: Frame IP not in any known module. Following frames may be wrong.
20 23bbeb74 21c67ff7     0x83e34b1
21 23bbebb4 21c67e3c     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.DequeueAndSend+0x5f
22 23bbebe4 6cabd4bb     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Channel.InMemoryTransmitter.Runner+0x54
23 23bbebf0 6cabb731     mscorlib_ni!System.Threading.Tasks.Task.InnerInvoke+0x4b
24 23bbec14 6cabb6fc     mscorlib_ni!System.Threading.Tasks.Task.Execute+0x31
25 23bbec7c 6ca58604     mscorlib_ni!System.Threading.Tasks.Task.ExecutionContextCallback+0x1c
26 23bbec7c 6ca58537     mscorlib_ni!System.Threading.ExecutionContext.RunInternal+0xc4
27 23bbec90 6cabb4b2     mscorlib_ni!System.Threading.ExecutionContext.Run+0x17
28 23bbecfc 6cabb357     mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal+0xe2
29 23bbed0c 6cad6d0a     mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry+0xb7
2a 23bbed1c 6ca32dd6     mscorlib_ni!System.Threading.Tasks.ThreadPoolTaskScheduler.LongRunningThreadWork+0x1e
2b 23bbed1c 6ca58604     mscorlib_ni!System.Threading.ThreadHelper.ThreadStart_Context+0x76
2c 23bbed80 6ca58537     mscorlib_ni!System.Threading.ExecutionContext.RunInternal+0xc4
2d 23bbed94 6ca584f4     mscorlib_ni!System.Threading.ExecutionContext.Run+0x17
2e 23bbedb0 6ca32d0e     mscorlib_ni!System.Threading.ExecutionContext.Run+0x44
2f 23bbedc8 6db4f066     mscorlib_ni!System.Threading.ThreadHelper.ThreadStart+0x52
30 23bbedd4 6db5234a     clr!CallDescrWorkerInternal+0x34
31 23bbee28 6db5862b     clr!CallDescrWorkerWithHandler+0x6b
32 23bbee9c 6dc69727     clr!MethodDescCallSite::CallTargetWorker+0x16a
33 (Inline) --------     clr!MethodDescCallSite::Call+0x12
34 23bbf00c 6dcfe376     clr!ThreadNative::KickOffThread_Worker+0x131
35 23bbf020 6dcfe401     clr!ManagedThreadBase_DispatchInner+0x71
36 (Inline) --------     clr!ManagedThreadBase_DispatchMiddle::__l16::Cleanup::{dtor}+0xfffffffc
37 23bbf0c4 6dcfe2f2     clr!ManagedThreadBase_DispatchMiddle+0x7e
38 23bbf118 6dd01655     clr!ManagedThreadBase_DispatchOuter+0x99
39 23bbf124 6dd015e1     clr!ManagedThreadBase_DispatchInCorrectAD+0x15
3a 23bbf1f0 6dd01680     clr!Thread::DoADCallBack+0x328
3b 23bbf214 6dcfe401     clr!ManagedThreadBase_DispatchInner+0x4e
3c (Inline) --------     clr!ManagedThreadBase_DispatchMiddle::__l16::Cleanup::{dtor}+0xfffffffc
3d 23bbf2b8 6dcfe2f2     clr!ManagedThreadBase_DispatchMiddle+0x7e
3e 23bbf30c 6dcfe4e1     clr!ManagedThreadBase_DispatchOuter+0x99
3f 23bbf330 6dc695d8     clr!ManagedThreadBase_FullTransitionWithAD+0x2f
40 23bbf3b4 6dc14c87     clr!ThreadNative::KickOffThread+0x260
41 23bbf9d8 7558fa29     clr!Thread::intermediateThreadProc+0x58
42 23bbf9e8 77497a7e     kernel32!BaseThreadInitThunk+0x19
43 23bbfa44 77497a4e     ntdll!__RtlUserThreadStart+0x2f
44 23bbfa54 00000000     ntdll!_RtlUserThreadStart+0x1b

Thread 5 => clr code => managed code => Entered lock => executing code => clr code => Waiting to enter critsec
https://github.com/microsoft/ApplicationInsights-dotnet/blob/317786702ee9c356f7e452b6739fcdcceb59f455/BASE/src/Microsoft.ApplicationInsights/Extensibility/Implementation/Tracing/DiagnosticsModule/DiagnoisticsEventThrottlingScheduler.cs#L48
https://github.com/microsoft/ApplicationInsights-dotnet/blob/317786702ee9c356f7e452b6739fcdcceb59f455/BASE/src/Microsoft.ApplicationInsights/Extensibility/Implementation/Tracing/CoreEventSource.cs#L111

0:005> kL
# ChildEBP RetAddr      
00 071d9c58 7748f909     ntdll!NtWaitForAlertByThreadId+0xc
01 071d9c58 7748f65d     ntdll!RtlpWaitOnAddressWithTimeout+0x64
02 (Inline) --------     ntdll!RtlpWaitOnAddress+0xbb
03 071d9cf8 774700ca     ntdll!RtlpWaitOnCriticalSection+0x18d
04 071d9d30 7746ff19     ntdll!RtlpEnterCriticalSectionContended+0x1aa
05 071d9d3c 6dc5b360     ntdll!RtlEnterCriticalSection+0x49
06 (Inline) --------     clr!UnsafeEnterCriticalSection+0x7
07 071d9d74 6dc5b2bb     clr!CrstBase::SpinEnter+0xd6
08 071d9d74 6db64b9c     clr!CrstBase::Enter+0x136
09 (Inline) --------     clr!Volatile<Thread *>::{ctor}+0xfffffffb
0a (Inline) --------     clr!VolatilePtr<Thread,Thread *>::{ctor}+0xfffffffb
0b (Inline) --------     clr!DeadlockAwareLock::EndEnterLock
0c (Inline) --------     clr!ListLockEntry::FinishDeadlockAwareEnter+0x1e
0d 071d9da4 6dc54e02     clr!ListLockEntry::LockHolder::DeadlockAwareAcquire+0x3e
0e 071da284 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x3e8
0f (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
10 071da30c 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
11 071da33c 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
WARNING: Frame IP not in any known module. Following frames may be wrong.
12 071da33c 21c6ac0d     0x83e34b1
13 071da354 21c6aa42     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingManager<System.__Canon>..ctor+0x6d
14 071da394 21c6a865     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule.Initialize+0xba
15 071da3c0 21c6a90a     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponent+0x35
16 071da3e8 21c6a815     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents+0x42
17 071da3f8 21c68e63     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents+0x4d
18 071da428 21c6851f     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize+0x22b
19 071da440 21c66445     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize+0x27
1a 071da470 21c65297     Microsoft_ApplicationInsights!Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.get_Active+0x95
1b 071da484 21c64e5b     Microsoft_ApplicationInsights_TraceListener!Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor+0x27
1c 071da494 6db4f066     Microsoft_ApplicationInsights_TraceListener!Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor+0xb
1d 071da494 6db5234a     clr!CallDescrWorkerInternal+0x34
1e 071da4e8 6db677f1     clr!CallDescrWorkerWithHandler+0x6b
1f 071da528 6db678f6     clr!CallDescrWorkerReflectionWrapper+0x55
20 071da82c 6ca85ce9     clr!RuntimeMethodHandle::InvokeMethod+0x838
21 071da854 2355b9f3     mscorlib_ni!System.Reflection.RuntimeConstructorInfo.Invoke+0x149
22 071da874 2355b421     System!System.SecurityUtils.ConstructorInfoInvoke+0x5b
23 071da8b0 2355b2d8     System!System.Diagnostics.TraceUtils.GetRuntimeObject+0xd9
24 071da8c8 2355afc1     System!System.Diagnostics.TypedElement.BaseGetRuntimeObject+0x30
25 071da8f4 2355a92e     System!System.Diagnostics.ListenerElement.GetRuntimeObject+0x1b1
26 071da928 2355a757     System!System.Diagnostics.ListenerElementsCollection.GetRuntimeObject+0xb6
27 071da958 6db4f066     System!System.Diagnostics.TraceInternal.get_Listeners+0x9f
28 071da964 6db5234a     clr!CallDescrWorkerInternal+0x34
29 071da9b8 6db5862b     clr!CallDescrWorkerWithHandler+0x6b
2a 071daa24 6e0b669c     clr!MethodDescCallSite::CallTargetWorker+0x16a
2b (Inline) --------     clr!MethodDescCallSite::CallWithValueTypes_RetArgSlot+0x6
2c 071daa70 6e0b6ba5     clr!FuncEvalWrapper+0x2e
2d 071dabc4 6e0b6f09     clr!DoNormalFuncEval+0x2b6
2e 071dad24 6e0b7c36     clr!GCProtectArgsAndDoNormalFuncEval+0x201
2f 071daed0 6e0b7dc1     clr!FuncEvalHijackRealWorker+0x796
30 071daf48 6db41186     clr!FuncEvalHijackWorker+0x14d
31 071daf5c 21c64374     clr!FuncEvalHijack+0x6
32 071daf8c 6d2542d2     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource.OnEventCommand+0x24
33 071dafd4 6ca6a986     mscorlib_ni!System.Diagnostics.Tracing.EventSource.DoCommand+0x21e
34 071db008 6ca4d87b     mscorlib_ni!System.Diagnostics.Tracing.EventSource.Initialize+0x352
35 071db03c 21c642e8     mscorlib_ni!System.Diagnostics.Tracing.EventSource..ctor+0x4f
36 071db050 21c64297     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource..ctor+0x28
37 071db050 6db4f066     System_Diagnostics_DiagnosticSource!System.Diagnostics.DiagnosticSourceEventSource..cctor+0x1f
38 071db05c 6db5234a     clr!CallDescrWorkerInternal+0x34
39 071db0b0 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
3a 071db0f4 6db64392     clr!DispatchCallDebuggerWrapper+0x59
3b 071db140 6dc54fde     clr!DispatchCallSimple+0x97
3c 071db1ac 6dc550ab     clr!MethodTable::RunClassInitEx+0xca
3d 071db690 6dc55216     clr!MethodTable::DoRunClassInitThrowing+0x45d
3e (Inline) --------     clr!SOIntolerantTransitionHandler::SetNoException
3f 071db718 6dc55280     clr!JIT_GetSharedNonGCStaticBase_Helper+0x85
40 071db75c 083e34b1     clr!JIT_GetSharedNonGCStaticBase_Portable+0x2f
41 071db75c 21c6412e     0x83e34b1
42 071db78c 21c6408e     Azure_Identity!+0x36
43 071db78c 21c63529     Azure_Identity!Azure.Core.Pipeline.ClientDiagnostics..ctor+0x4e
44 071db7a4 21c62437     Azure_Identity!+0x71
45 071db7b4 6ca7c49b     Azure_Identity!+0x2f
46 071db7e0 6ca7c10f     mscorlib_ni!System.Lazy`1.CreateValue+0xbb
47 071db814 6ca7c040     mscorlib_ni!System.Lazy`1.LazyInitValue+0x9f
48 071db824 21c61b83     mscorlib_ni!System.Lazy`1.get_Value+0x50
49 071db834 21c621d6     Azure_Identity!+0x43
4a 071db848 21c616cf     Azure_Identity!Azure.Identity.DefaultAzureCredential..ctor+0x26
4b 071db884 21c605ac     Microsoft_Configuration_ConfigurationBuilders_Azure!Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder.LazyInitialize+0x1b7
4c 071db8b4 21c60646     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureInitialized+0x5c
4d 071db8e4 21c60fa0     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.GetValueInternal+0x4e
4e 071db928 21c7686d     Microsoft_Configuration_ConfigurationBuilders_Base!Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.ProcessConfigurationSection+0xd0
4f 071db930 21c7689b     System_Configuration!System.Configuration.Internal.InternalConfigHost.System.Configuration.Internal.IInternalConfigurationBuilderHost.ProcessConfigurationSection+0x15
50 071db93c 21c76973     System_Configuration!System.Configuration.Internal.DelegatingConfigHost.ProcessConfigurationSection+0x13
51 071db968 21b2f5b8     System_Configuration!System.Configuration.BaseConfigurationRecord.CallHostProcessConfigurationSection+0x33
52 071db984 21b85152     System_Configuration!System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionImpl+0xc8
53 071db9c8 21b2ec7a     System_Configuration!System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionWithRestrictedPermissions+0x62
54 071db9f0 21b2eb24     System_Configuration!System.Configuration.RuntimeConfigurationRecord.CreateSection+0x62
55 071dba3c 21b341b8     System_Configuration!System.Configuration.BaseConfigurationRecord.CallCreateSection+0x6c
56 071dba74 21b33f1f     System_Configuration!System.Configuration.BaseConfigurationRecord.EvaluateOne+0x58
57 071dbb34 21b2df56     System_Configuration!System.Configuration.BaseConfigurationRecord.Evaluate+0x43f
58 071dbbb4 21b2db23     System_Configuration!System.Configuration.BaseConfigurationRecord.GetSectionRecursive+0x416
59 071dbbdc 21bafe86     System_Configuration!System.Configuration.BaseConfigurationRecord.GetSection+0x2b
5a 071dbbec 21c71339     System_Web!System.Web.Util.AppSettings.GetAppSettingsSection+0x56
5b 071dbcf4 21c720a7     System_Web!System.Web.Util.AppSettings.EnsureSettingsLoaded+0x89
5c 071dbd08 21c722b4     System_Web!System.Web.Compilation.BuildManager.RestorePortableCompilationOutputSnapshot+0x1f
5d 071dbd38 085b3098     System_Web!System.Web.Compilation.BuildManager.ExecutePreAppStart+0x14
5e 071dbd70 085b2026     System_Web!System.Web.Hosting.HostingEnvironment.Initialize+0x2b8
5f 071dbd8c 6db4f066     System_Web!System.Web.Hosting.HostingEnvironment.Initialize+0x16
60 071dbda8 6db5234a     clr!CallDescrWorkerInternal+0x34
61 071dbdfc 6dc54ed4     clr!CallDescrWorkerWithHandler+0x6b
62 071dbe40 6dc1cbed     clr!DispatchCallDebuggerWrapper+0x59
63 071dbea0 6dc1dd92     clr!DispatchCall+0x7d
64 (Inline) --------     clr!RemotableMethodInfo::IsReturnGCRef+0xfffffff7
65 071dc048 6dc1ecaf     clr!CrossDomainChannel::MarshalAndCall_Wrapper+0x4c9
66 071dc054 6de314fe     clr!MarshalAndCall_Wrapper2+0xf
67 071dc09c 6de3158c     clr!MakeADCallDebuggerWrapper+0x5f
68 071dc124 6dc1eae6     clr!MakeCallWithAppDomainTransition+0x8f
69 071dd5cc 6dc1cccf     clr!CrossDomainChannel::MarshalAndCall+0x7a8
6a 071dd5f4 6dc1cdbb     clr!CrossDomainChannel::ExecuteCrossDomainCall+0x60
6b 071dd608 6dc1ce7e     clr!CrossDomainChannel::CheckCrossDomainCall+0xa8
6c 071dd728 6db4f566     clr!TransparentProxyStubWorker+0xbd
6d 071dd758 6a75b298     clr!TransparentProxyStub_CrossContext+0x14
6e 071dda24 6a759986     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment+0x18a4
6f 071dda54 6a7598f5     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors+0x26
70 071dda98 6a759843     System_Web_ni!System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment+0x6d
71 071ddab8 6a758ef5     System_Web_ni!System.Web.Hosting.ApplicationManager.CreateObjectInternal+0x4f
72 071ddb0c 6adaaaab     System_Web_ni!System.Web.Hosting.ProcessHost.StartApplication+0x195
73 071ddb60 6db4f434     System_Web_ni+0x82aaab
74 071ddb88 6dc14143     clr!COMToCLRDispatchHelper+0x6b
75 (Inline) --------     clr!InvokeStub+0x3e
76 (Inline) --------     clr!COMToCLRInvokeTarget+0x5b
77 (Inline) --------     clr!COMToCLRWorkerBody+0x14f
78 071ddbe0 083ed085     clr!COMToCLRWorker+0x4d5
79 071ddc0c 67517444     CLRStub[StubLinkStub]@140a320d083ed085
7a 071ddc5c 6750bc76     webengine4!W3_MGD_APP_CONTEXT::CreateAppDomain+0xbb
7b 071ddc7c 6750bd3e     webengine4!W3_MGD_APP_CONTEXT::OnApplicationResolveModules+0xa9
7c 071ddc94 798071cb     webengine4!CMgdEngGlobalModule::OnGlobalApplicationResolveModules+0x5d
7d 071ddca8 797eb767     iiscore!VIRTUAL_MODULE::GlobalDoWork+0x15b
7e 071ddcd8 79808249     iiscore!W3_SERVER::GlobalNotify+0x115
7f 071ddd10 7980830f     iiscore!W3_APPLICATION::ResolveModules+0x22
80 071ddd10 797f0f8e     iiscore!W3_APPLICATION::SetupNotificationContext+0x86
81 071ddd58 797f0686     iiscore!W3_CONTEXT::SetupStateMachinePhase2+0x3e2
82 071dfddc 797f8607     iiscore!W3_CONTEXT::SetupStateMachine+0x2b8
83 071dfdf4 797f8b24     iiscore!W3_CONTEXT_BASE::StartNotificationLoop+0x72
84 071dfe0c 79836d9f     iiscore!W3_MAIN_CONTEXT::OnNewRequest+0xb4
85 (Inline) --------     w3dt!UL_NATIVE_REQUEST::DoStateProcess+0x17
86 071dfe28 79837b8b     w3dt!UL_NATIVE_REQUEST::DoWork+0xa6
87 071dfe40 7983638f     w3dt!UL_RECEIVE_CONTEXT::DoWork+0x1b
88 071dfe5c 7982329b     w3dt!WP_CONTEXT::OnCompletion+0x2f
89 071dfe90 798231bb     w3tp!THREAD_POOL_DATA::ThreadPoolThread+0xa2
8a 071dfeb0 798220fc     w3tp!THREAD_POOL_DATA::ThreadPoolThread+0x2b
8b 071dfecc 7558fa29     w3tp!THREAD_MANAGER::ThreadManagerThread+0x5c
8c 071dfedc 77497a7e     kernel32!BaseThreadInitThunk+0x19
8d 071dff38 77497a4e     ntdll!__RtlUserThreadStart+0x2f
8e 071dff48 00000000     ntdll!_RtlUserThreadStart+0x1b
0:005> !CLRStack
OS Thread Id: 0x5e80 (5)
Child SP       IP Call Site
071da2a0 774a46bc [HelperMethodFrame: 071da2a0] 
071da32c 21c6acea Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingScheduler.ScheduleToRunEveryTimeIntervalInMilliseconds(Int32, System.Action)
071da348 21c6ac0d Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.DiagnoisticsEventThrottlingManager`1[[System.__Canon, mscorlib]]..ctor(System.__Canon, Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsModule.IDiagnoisticsEventThrottlingScheduler, UInt32)
071da364 21c6aa42 Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da39c 21c6a865 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponent(System.Object, Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da3c8 21c6a90a Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents(System.Collections.IEnumerable, Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration)
071da3f0 21c6a815 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.InitializeComponents(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules)
071da400 21c68e63 Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules, System.String)
071da438 21c6851f Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize(Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration, Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryModules)
071da44c 21c66445 Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.get_Active()
071da478 21c65297 Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor(System.String)
071da48c 21c64e5b Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener..ctor()
071da7a0 6db4f066 [DebuggerU2MCatchHandlerFrame: 071da7a0] 
071da560 6db4f066 [HelperMethodFrame_PROTECTOBJ: 071da560] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean)
071da83c 6ca85ce9 System.Reflection.RuntimeConstructorInfo.Invoke(System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
071da868 2355b9f3 System.SecurityUtils.ConstructorInfoInvoke(System.Reflection.ConstructorInfo, System.Object[])
071da87c 2355b421 System.Diagnostics.TraceUtils.GetRuntimeObject(System.String, System.Type, System.String)
071da8bc 2355b2d8 System.Diagnostics.TypedElement.BaseGetRuntimeObject()
071da8d0 2355afc1 System.Diagnostics.ListenerElement.GetRuntimeObject()
071da8fc 2355a92e System.Diagnostics.ListenerElementsCollection.GetRuntimeObject()
071da930 2355a757 System.Diagnostics.TraceInternal.get_Listeners()
071dab78 6db4f066 [GCFrame: 071dab78] 
071dab5c 6db4f066 [GCFrame: 071dab5c] 
071dace0 6db4f066 [ProtectValueClassFrame: 071dace0] 
071dac30 6db4f066 [GCFrame: 071dac30] 
071dac14 6db4f066 [GCFrame: 071dac14] 
071dabf8 6db4f066 [GCFrame: 071dabf8] 
071daf08 6db4f066 [FuncEvalFrame: 071daf08] 
071daf54 21c6430e System.Diagnostics.DiagnosticSourceEventSource.BreakPointWithDebuggerFuncEval()
071daf64 21c64374 System.Diagnostics.DiagnosticSourceEventSource.OnEventCommand(System.Diagnostics.Tracing.EventCommandEventArgs)
071daf94 6d2542d2 System.Diagnostics.Tracing.EventSource.DoCommand(System.Diagnostics.Tracing.EventCommandEventArgs)
071dafdc 6ca6a986 System.Diagnostics.Tracing.EventSource.Initialize(System.Guid, System.String, System.String[])
071db024 6ca4d87b System.Diagnostics.Tracing.EventSource..ctor(System.Diagnostics.Tracing.EventSourceSettings, System.String[])
071db048 21c642e8 System.Diagnostics.DiagnosticSourceEventSource..ctor()
071db04c 21c64297 System.Diagnostics.DiagnosticSourceEventSource..cctor()
071db1cc 6db4f066 [GCFrame: 071db1cc] 
071db6ac 6db4f066 [HelperMethodFrame: 071db6ac] 
071db738 21c64207 System.Diagnostics.DiagnosticListener..ctor(System.String)
071db764 21c6412e Azure.Core.Pipeline.DiagnosticScopeFactory..ctor(System.String, System.String, Boolean)
071db77c 21c6408e Azure.Core.Pipeline.ClientDiagnostics..ctor(Azure.Core.ClientOptions)
071db794 21c63529 Azure.Identity.CredentialPipeline..ctor(Azure.Identity.TokenCredentialOptions)
071db7ac 21c62437 Azure.Identity.CredentialPipeline+c.b__19_0()
071db7bc 6ca7c49b System.Lazy`1[[System.__Canon, mscorlib]].CreateValue()
071db7e8 6ca7c10f System.Lazy`1[[System.__Canon, mscorlib]].LazyInitValue()
071db81c 6ca7c040 System.Lazy`1[[System.__Canon, mscorlib]].get_Value()
071db82c 21c61b83 Azure.Identity.CredentialPipeline.GetInstance(Azure.Identity.TokenCredentialOptions)
071db83c 21c621d6 Azure.Identity.DefaultAzureCredential..ctor(Azure.Identity.DefaultAzureCredentialOptions)
071db850 21c616cf Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder.LazyInitialize(System.String, System.Collections.Specialized.NameValueCollection)
071db890 21c605ac Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureInitialized()
071db8bc 21c60646 Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.GetValueInternal(System.String)
071db8ec 21c60fa0 Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.ProcessConfigurationSection(System.Configuration.ConfigurationSection)
071db930 21c7686d System.Configuration.Internal.InternalConfigHost.System.Configuration.Internal.IInternalConfigurationBuilderHost.ProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db93c 21c7689b System.Configuration.Internal.DelegatingConfigHost.ProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db948 21c76973 System.Configuration.BaseConfigurationRecord.CallHostProcessConfigurationSection(System.Configuration.ConfigurationSection, System.Configuration.ConfigurationBuilder)
071db974 21b2f5b8 System.Configuration.RuntimeConfigurationRecord+RuntimeConfigurationFactory.CreateSectionImpl(System.Configuration.RuntimeConfigurationRecord, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071db9a0 21b85152 System.Configuration.RuntimeConfigurationRecord+RuntimeConfigurationFactory.CreateSectionWithRestrictedPermissions(System.Configuration.RuntimeConfigurationRecord, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071db9e4 21b2ec7a System.Configuration.RuntimeConfigurationRecord.CreateSection(Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071dba0c 21b2eb24 System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Configuration.SectionInput, System.Object, System.Configuration.ConfigXmlReader)
071dba58 21b341b8 System.Configuration.BaseConfigurationRecord.EvaluateOne(System.String[], System.Configuration.SectionInput, Boolean, System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Object)
071dba90 21b33f1f System.Configuration.BaseConfigurationRecord.Evaluate(System.Configuration.FactoryRecord, System.Configuration.SectionRecord, System.Object, Boolean, Boolean, System.Object ByRef, System.Object ByRef)
071dbb54 21b2df56 System.Configuration.BaseConfigurationRecord.GetSectionRecursive(System.String, Boolean, Boolean, Boolean, Boolean, System.Object ByRef, System.Object ByRef)
071dbbd4 21b2db23 System.Configuration.BaseConfigurationRecord.GetSection(System.String)
071dbbe4 21bafe86 System.Web.Util.AppSettings.GetAppSettingsSection()
071dbbf4 21c71339 System.Web.Util.AppSettings.EnsureSettingsLoaded()
071dbcfc 21c720a7 System.Web.Compilation.BuildManager.RestorePortableCompilationOutputSnapshot()
071dbd10 21c722b4 System.Web.Compilation.BuildManager.ExecutePreAppStart()
071dbd40 085b3098 System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel, System.Exception)
071dbd8c 085b2026 System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
071dbf18 6db4f066 [GCFrame: 071dbf18] 
071dbefc 6db4f066 [GCFrame: 071dbefc] 
071dc0bc 6db4f066 [ContextTransitionFrame: 071dc0bc] 
071dc218 6db4f066 [GCFrame: 071dc218] 
071dc1fc 6db4f066 [GCFrame: 071dc1fc] 
071dd4d4 6db4f066 [GCSafeCollectionFrame: 071dd4d4] 
071dd4e4 6db4f066 [GCSafeCollectionFrame: 071dd4e4] 
071dd4f4 6db4f066 [GCSafeCollectionFrame: 071dd4f4] 
071dd504 6db4f066 [GCSafeCollectionFrame: 071dd504] 
071dd514 6db4f066 [GCSafeCollectionFrame: 071dd514] 
071dd524 6db4f066 [GCSafeCollectionFrame: 071dd524] 
071dd534 6db4f066 [GCSafeCollectionFrame: 071dd534] 
071dd704 6db4f066 [TPMethodFrame: 071dd704] System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
071dd770 6a75b298 System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071dda34 6a759986 System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071dda64 6a7598f5 System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
071ddaa8 6a759843 System.Web.Hosting.ApplicationManager.CreateObjectInternal(System.String, System.Type, System.Web.Hosting.IApplicationHost, Boolean, System.Web.Hosting.HostingEnvironmentParameters)
071ddad0 6a758ef5 System.Web.Hosting.ProcessHost.StartApplication(System.String, System.String, System.Object ByRef)
071ddb1c 6adaaaab DomainNeutralILStubClass.IL_STUB_COMtoCLR(IntPtr, IntPtr, IntPtr)
071ddbf4 6db4f434 [ComMethodFrame: 071ddbf4]

Author: TimothyMothra
Assignees: -
Labels:

area-System.Diagnostics.Tracing, untriaged

Milestone: -

@noahfalk noahfalk removed the untriaged New issue has not been triaged by the area owner label Jul 15, 2021
@noahfalk noahfalk self-assigned this Jul 15, 2021
@noahfalk
Copy link
Member

Thanks for the nice writeup @TimothyMothra !

@gregg-miskelly - who is good person for handling func-eval issues these days?
I think the short summary of the issue is that we've got a deadlock with this cycle:
thread 5: Enter event source monitor lock -> hit DiagnosticSourceEventSource.BreakpointWithDebuggerFuncEval() -> VS ran funceval -> funceval blocked on internal runtime critical section for running AI's CoreEventSource..cctor
thread 48: Acquire internal runtime lock for AI's CoreEventSource..cctor -> block acquiring event source monitor lock

When thread 5 tries to abort the func-eval I assume graceful abort fails because the thread is holding the event source monitor lock. In general that placement for BreakpointWithDebuggerFuncEval() is awkward because it means every func-eval run from that point will always have a monitor lock held. One potential solution would be don't func-eval at that point and instead step-out three frames which will get the app clear of the lock, then do the func-eval.

@noahfalk
Copy link
Member

I marked this as 7.0, but realistically the .NET milestone doesn't matter for issues resolved in Visual Studio. If it does wind up being something that needs a .NET change we'll need to evaluate what work is involved.

@delmyers
Copy link
Contributor

So, we've found a work-around for this issue inside VS. I'm working on a fix right now. As things go, it would probably be available in VS 2022. I don't know if we have a path to getting it fixed in 2019, though.

@delmyers
Copy link
Contributor

delmyers commented Aug 3, 2021

This issue has been fixed. It will be released with VS 2022, Preview 4. If there is a large need to have it in VS 2019, we can see about back porting the fix. Let us know.

@haldiggs
Copy link

haldiggs commented Aug 3, 2021

I know I would love to see it get into 2019 only because after 2019 I will not be able to use the load test and things that I use with my WCF project. Therefore I will be developing in 2019 for quite a while at this point. We are unable to divert resources to rewrite our custom load testing. So anyway the trace timeouts really would be something that would be nice not to have to worry about

@noahfalk
Copy link
Member

noahfalk commented Aug 3, 2021

Thanks @delmyers much appreciated!

@noahfalk noahfalk modified the milestones: 7.0.0, Future Aug 3, 2021
@argelj289
Copy link

argelj289 commented Aug 4, 2021

Looking forward to include the fix in VS 2019 as well

@fowl2
Copy link

fowl2 commented Aug 4, 2021

As long as the workaround won't break in VS 2021 I'm fine with no backports. (Is there a way to conditionalize project file properties on VS version?)

@delmyers
Copy link
Contributor

@argelj289 @haldiggs OK, we have fixed the issue in VS 2019 as well, version 16.11.2. It should be released in the next month or two as our servicing releases come out.

@haldiggs
Copy link

haldiggs commented Aug 19, 2021 via email

@noahfalk
Copy link
Member

@delmyers @TimothyMothra - is there anything left that needs fixing being tracked by this issue or can we close it?

@delmyers
Copy link
Contributor

@noahfalk Nothing that I know of. You can go ahead and close this as far as I'm concerned.

@TimothyMothra
Copy link
Author

My team hasn't received any customer complaints since the fix was released. I think we can close this. :)

@ghost ghost locked as resolved and limited conversation to collaborators Aug 25, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants