-
Notifications
You must be signed in to change notification settings - Fork 6.8k
[BUGFIX] Free up executor memory when reassigning new executor #19214
Conversation
Hey @access2rohit , Thanks for submitting the PR
CI supported jobs: [clang, windows-cpu, centos-gpu, website, windows-gpu, sanity, unix-cpu, edge, centos-cpu, unix-gpu, miscellaneous] Note: |
Looks like we only instantiate the object the first time because we want to check whether subgraph property is set, which has nothing to do with the executor really. We can define a variable like this and use it in the condition: https://github.com/apache/incubator-mxnet/blob/07e65e5cf3c629c78f206798f33a3529e95c0fc4/src/executor/graph_executor.cc#L58 |
07e65e5
to
3b0b9dc
Compare
@mseth10 Can you review again ? |
src/executor/graph_executor.cc
Outdated
if (!exec->subgraph_property().empty()) { | ||
const auto& backend_name = exec->subgraph_property(); | ||
const auto& backend_name = dmlc::GetEnv("MXNET_SUBGRAPH_BACKEND", exec::GetDefaultSubgraphBackend()); | ||
if (!backend_name.empty()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should also include the check backend_name == "None"
or we can do something like this before the condition https://github.com/apache/incubator-mxnet/blob/3b0b9dc9d865ee0b1456e8f400f6f4a25c77b5db/src/executor/graph_executor.cc#L59-L60
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer we dont duplicate code/functionality if at all possible. Since we create exec
with new
, and then use it to get the backend_name
, why dont we just call delete on exec
before line 2080 when creating a new executor from the partitioned symbol?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah I agree, we can just use delete exec
for both functions
We also need to make this change for the SimpleBind API, it is implemented in a similar fashion https://github.com/apache/incubator-mxnet/blob/3b0b9dc9d865ee0b1456e8f400f6f4a25c77b5db/src/executor/graph_executor.cc#L2009 |
3b0b9dc
to
9c9baaf
Compare
9c9baaf
to
2cbef0a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the delete change, this looks very succinct!
Description
Fixes memory leak caused due to not freeing up of graph executor object.
Checklist
Essentials