Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coordinate shutdown of AdaptiveDirectoryCacheMaintainer with LocalGrainDirectory #9061

Conversation

ReubenBond
Copy link
Member

@ReubenBond ReubenBond commented Jul 11, 2024

I saw this during a reliability test run:

[08:41:39 INF] fail: Orleans.Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer[100026]
[08:41:39 INF]       Asynch agent Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer encountered unexpected exception. The Stage will be restarted.
[08:41:39 INF]       System.ObjectDisposedException: Cannot access a disposed object.
[08:41:39 INF]       Object name: 'IServiceProvider'.
[08:41:39 INF]          at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
[08:41:39 INF]          at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
[08:41:39 INF]          at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
[08:41:39 INF]          at Orleans.GrainReferences.GrainReferenceActivatorProvider.TryGet(GrainType grainType, GrainInterfaceType interfaceType, IGrainReferenceActivator& activator) in C:\dev\orleans\src\Orleans.Core\GrainReferences\GrainReferenceActivator.cs:line 336
[08:41:39 INF]          at Orleans.GrainReferences.GrainReferenceActivator.CreateActivator(GrainType grainType, GrainInterfaceType interfaceType) in C:\dev\orleans\src\Orleans.Core\GrainReferences\GrainReferenceActivator.cs:line 77
[08:41:39 INF]          at Orleans.GrainReferences.GrainReferenceActivator.CreateReference(GrainId grainId, GrainInterfaceType interfaceType) in C:\dev\orleans\src\Orleans.Core\GrainReferences\GrainReferenceActivator.cs:line 54
[08:41:39 INF]          at Orleans.GrainFactory.CreateGrainReference(Type interfaceType, GrainId grainId) in C:\dev\orleans\src\Orleans.Core\Core\GrainFactory.cs:line 230
[08:41:39 INF]          at Orleans.GrainFactory.GetGrain[TGrainInterface](GrainId grainId) in C:\dev\orleans\src\Orleans.Core\Core\GrainFactory.cs:line 144
[08:41:39 INF]          at Orleans.GrainFactory.GetSystemTarget[TGrainInterface](GrainId grainId) in C:\dev\orleans\src\Orleans.Core\Core\GrainFactory.cs:line 135
[08:41:39 INF]          at Orleans.GrainFactory.GetSystemTarget[TGrainInterface](GrainType grainType, SiloAddress destination) in C:\dev\orleans\src\Orleans.Core\Core\GrainFactory.cs:line 118
[08:41:39 INF]          at Orleans.Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer.SendBatchCacheRefreshRequests(Dictionary`2 refreshRequests) in C:\dev\orleans\src\Orleans.Runtime\GrainDirectory\AdaptiveDirectoryCacheMaintainer.cs:line 136
[08:41:39 INF]          at Orleans.Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer.Run() in C:\dev\orleans\src\Orleans.Runtime\GrainDirectory\AdaptiveDirectoryCacheMaintainer.cs:line 116
[08:41:39 INF]          at Orleans.Runtime.TaskSchedulerAgent.StartAsync() in C:\dev\orleans\src\Orleans.Core\Scheduler\TaskSchedulerAgent.cs:line 77

This PR fixes the error by coordinating the shutdown of AdaptiveDirectoryCacheMaintainer with LocalGrainDirectory and Silo.

Microsoft Reviewers: Open in CodeFlow

@ReubenBond ReubenBond force-pushed the fix/adaptivedirectorycachemaintainer-lifecycle branch 3 times, most recently from 663346e to b08d1c9 Compare July 11, 2024 16:22
@ReubenBond ReubenBond force-pushed the fix/adaptivedirectorycachemaintainer-lifecycle branch from b08d1c9 to 5949e6b Compare July 11, 2024 16:32
@ReubenBond ReubenBond merged commit a264ce0 into dotnet:main Jul 11, 2024
18 of 22 checks passed
@ReubenBond ReubenBond deleted the fix/adaptivedirectorycachemaintainer-lifecycle branch July 11, 2024 17:11
@github-actions github-actions bot locked and limited conversation to collaborators Aug 11, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants