-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Very Large CPU RAM Memory Consumption (>1GB) #15148
Comments
Hey, this is the MXNet Label Bot. |
@mxnet-label-bot Add [Performance, Memory] |
@rvardimon thanks for raising the issue. Have you looked at the profiler output from mxnet , when you load the model params it initially loads on cpu so you may see a spike but then it should be released after set_params is called. if not module api is adding some overhead which should be looked at: @karan6181 |
|
We ( I and @karan6181 ) looked at a bunch of things to understand where the overhead is coming from. We looked at Resource Request and attach op resources pass. We looked at object pool in threaded engine, and at turning off OPENMP and MKLDNN and checking, but the overhead is still there. We looked at the overhead caused by different arrays in the module api. Overhead is not coming from any of these areas. We also check that increase in memory consumption happens at the bind stage and probably coming from somewhere in the graph executor. Next step is to check this in 1.2.1 to see if the increase happened in 1.3.1. |
I used mxnet-cu100mkl 1.4.1 in win10. |
@anirudh2290 Hi did you discover any solution for this memory consumption issue ? Best |
Description
Mxnet consumes nearly 2GB CPU RAM even when loading a relatively small model (e.g. Resnet-18) directed on GPU (
ctx=mxnet.gpu()
). From what I understand, there is no real need to use so much CPU memory when the model is running on GPU.This issue is extremely prohibitive when trying to run multiple processes with mxnet on the same machine, and IMO gives it a significant disadvantage compared to other frameworks for being used in AI production systems.
Environment info
Package used (Python/R/Scala/Julia):
I'm using Python3
Build info
mxnet installed using pip3
Steps to reproduce
(Paste the commands you ran that produced the error.)
The text was updated successfully, but these errors were encountered: