resmgr: lifecycle overlap detection and workaround. #358
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use PrettyName()-compatible container name dumps together with container IDs in NRI evnet dumps. Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
Having loggers embedded in resmgr types as members was a bad idea. Replace them with module global logger instances. Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
klihub
force-pushed
the
fixes/overlapping-lifecycle-workaround
branch
from
0d2f646
to
c4b6aba
Compare
We recently discovered a problem with the generated stream of container lifecycle events with some runtime versions. A side effect of this is that we get Create/Stop events for multiple container instances with seemingly overlapping lifecycle: the latter instance get created before the former one is stopped. When undetected, such a false overlap might cause overcommit of resources, with both instances temporarily using the full resource set of the container. As a workaround, we now track containers also by fully qualified name ($namespace/pod/ctr) and internally generate an event for releasing the resources if the old instance whenever we notice that a creation event would cause a duplicate instance for the same name. Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
klihub
force-pushed
the
fixes/overlapping-lifecycle-workaround
branch
from
c4b6aba
to
d551afc
Compare
9 tasks
klihub
force-pushed
the
fixes/overlapping-lifecycle-workaround
branch
from
f609d93
to
934044d
Compare
kad
reviewed
Sep 20, 2024
pfl
approved these changes
Sep 20, 2024
Check grants, looking for grants with stale allocations or duplicate containers (detected using fully qualified names). Dump total memory and CPU granted. Signed-off-by: Krisztian Litkey <krisztian.litkey@intel.com>
klihub
force-pushed
the
fixes/overlapping-lifecycle-workaround
branch
from
934044d
to
63e4cdd
Compare
kad
approved these changes
Sep 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We recently discovered a problem with the generated stream of container lifecycle events with some runtime versions. A side effect of this is that we get Create/Stop events for multiple container instances with seemingly overlapping lifecycle: the latter instance get created before the former one is stopped.
When undetected, such a false overlap might cause overcommit of resources, with both instances temporarily using the full resource set of the offending container. As a workaround, we now track containers also by fully qualified name ($namespace/pod/ctr) and internally generate an event for releasing the resources of the old instance whenever we notice that a creation event would cause a duplicate instance for the same name.