Provide extended per-process memory stats

[giampaolo]

From g.rodola on April 23, 2012 21:24:55

See ml discussion at: 
https://groups.google.com/forum/?fromgroups#!topic/psutil/Yepd8QNTKF0 As of 
right now psutil is only able to return RSS (resident set size) and VMS 
(virtual memory size) memory stats about a given process.

How the process memory is handled and under what values it is represented 
depends on what platform we're on though, therefore RSS/VMS  alone are not 
enough to represent process memory usage exhaustively.

Under Windows, for example, we have different values we are currently not 
returning: 
http://msdn.microsoft.com/en-us/library/windows/desktop/ms684877(v=vs.85).aspx 
Same thing for UNIX where we have "shared mem", "library mem", "fault pages" 
and others, depending on the UNIX flavor.

My proposal is to add a new Process.get_ext_memory_info() method returning a 
namedtuple whose attributes depend on what platform we're on. Alternatively 
Process.get_memory_info() can grown an extra "extended" argument defaulting to False.

Original issue: http://code.google.com/p/psutil/issues/detail?id=261

[giampaolo]

From wj32...@gmail.com on April 23, 2012 15:59:24

Unfortunately retrieving the private working set can be a bit slow on pre-Vista 
systems, since it involves calling QueryWorkingSet and summing up the private pages.

Do you have a list of what attributes you're looking for?

[giampaolo]

From g.rodola on April 24, 2012 06:25:53

I'm not a Windows guy so I'm not sure what memory stats are more useful than 
others on Windows.
If private pages are the same thing as on Linux then they may be important (see 
issue 96 ), but for example on Linux I plan to provide "shared" and "private" 
memory stats by iterating over process's shared libraries, as in:


total_pws = 0
for lib in p.get_shared_libs():
    total_pws += lib.private_ws


...instead of in get_memory_info().
I suppose this is not the case on Windows as private working set is not bound 
to process shared libs/modules, am I right?
In this case, yes, it would be good to have the private ws as well.
Are you aware of other memory stats which might be worth including?

[giampaolo]

From wj32...@gmail.com on April 24, 2012 19:10:07

All the stats shown by PE or PH can be useful.

[giampaolo]

From sharpbla...@gmail.com on April 25, 2012 07:55:18

"Unfortunately retrieving the private working set can be a bit slow on 
pre-Vista systems, since it involves calling QueryWorkingSet and summing up the 
private pages."

Windows XP was released in 2001, lets not let it stop us from adding new features.

[giampaolo]

From g.rodola on April 25, 2012 08:10:52

Indeed.  On Windows <= XP it would be ok to fallback on using QueryWorkingSet 
or if that's too much work it's also ok to force that specific field to "-1".

[giampaolo]

From g.rodola on July 08, 2012 13:11:23

Linux patch in attachment.

Attachment: linux-extmem.patch

[giampaolo]

From g.rodola on July 10, 2012 13:30:26

Fixed in r1422 (Linux), r1424 (OSX), r1425 (BSD) and r1426 (Windows).
The private working set on Windows is ignored for now.
Perhaps I'll add that later once I figure out a reliable way to check its 
availability at compile time.

Status: FixedInSVN
Labels: Milestone-1.0.0

[giampaolo]

From g.rodola on July 10, 2012 16:11:20

PrivateUsage on Windows committed as r1427 .

[giampaolo]

From g.rodola on August 13, 2012 09:25:13

Fixed in version 0.6.0, released just now.

Status: Fixed
Labels: -Milestone-1.0.0 Milestone-0.6.0

[giampaolo]

From g.rodola on March 02, 2013 04:07:34

Updated csets after the SVN -> Mercurial migration: r1422 == revision 
40f98485c267 r1427 == revision cc26dd332ce7