Take advantage of the PSUtil issue #1025 (Add process_iter(attrs, ad_value))

[nicolargo]

The PsUtil issue giampaolo/psutil#1025 (available in PsUtil 5.3 and higher) will simplify the way that Glances iters through the process.

See __get_process_stats function in glances/processes.py

[nicolargo]

PsUtil 5.3 is not available for the moment. Will be implemented in Glances 3.0.

[nicolargo]

See PsUtil documentation here: https://psutil.readthedocs.io/en/latest/#psutil.process_iter

[nicolargo]

Merged in the latest commit on the DEVELOP branch.

@asergi @notFloran : Can you test on your system ? It is a relative big refactoring of the Glances processes plugin...

[oppianmatt]

Is there a docker label for the develop branch? i.e. could I do something like this to test this out?

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock:ro --pid host --network host -it docker.io/nicolargo/glances:develop

[nicolargo]

Tags list is available here: https://hub.docker.com/r/nicolargo/glances/tags/

So latest (default) image is generated from the DEVELOP branch.

Stable from the MASTER.

[notFloran]

First try on macOS :

Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/floranbrutel/dev/glances/glances/__main__.py", line 29, in <module>
    glances.main()
  File "glances/__init__.py", line 140, in main
    start(config=config, args=args)
  File "glances/__init__.py", line 106, in start
    mode = GlancesMode(config=config, args=args)
  File "glances/webserver.py", line 41, in __init__
    self.stats.update()
  File "glances/stats.py", line 221, in update
    self._plugins[p].update()
  File "/Users/floranbrutel/dev/glances/glances/plugins/glances_processcount.py", line 55, in update
    glances_processes.update()
  File "glances/processes.py", line 418, in update
    key=lambda p: p.info['cpu_percent'])]
  File "/Users/floranbrutel/glances-venv/lib/python2.7/site-packages/psutil/__init__.py", line 1437, in process_iter
    yield add(pid)
  File "/Users/floranbrutel/glances-venv/lib/python2.7/site-packages/psutil/__init__.py", line 1419, in add
    proc.info = proc.as_dict(attrs=attrs, ad_value=ad_value)
  File "/Users/floranbrutel/glances-venv/lib/python2.7/site-packages/psutil/__init__.py", line 517, in as_dict
    ", ".join(map(repr, invalid_names))))
ValueError: invalid attr name 'io_counters'

[nicolargo]

Forget it, i will create a new specific branch and restart the dev from scratch. The current implementation is not good...

[nicolargo]

Ok merged in the DEVELOP branch:

@asergi and @notFloran : Can you test it ?

[notFloran]

On macOS :

$ LANGUAGE=en_US.utf8  ~/glances-venv/bin/python -m glances -d
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.14_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/floranbrutel/dev/glances/glances/__main__.py", line 29, in <module>
    glances.main()
  File "glances/__init__.py", line 140, in main
    start(config=config, args=args)
  File "glances/__init__.py", line 106, in start
    mode = GlancesMode(config=config, args=args)
  File "glances/standalone.py", line 68, in __init__
    self.stats.update()
  File "glances/stats.py", line 221, in update
    self._plugins[p].update()
  File "/Users/floranbrutel/dev/glances/glances/plugins/glances_processcount.py", line 55, in update
    glances_processes.update()
  File "glances/processes.py", line 276, in update
    self.update_processcount(self.processlist)
  File "glances/processes.py", line 99, in update_processcount
    self.processcount['thread'] = sum(i['num_threads'] for i in plist)
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

[nicolargo]

@notFloran Should be ok. can you test it (again) ?

[notFloran]

List and sorts works without issues but the filtering system is ko :

[nicolargo]

@notFloran can not reproduce on my system... Filter works like a charm... Anything relevant in the debug log file ?

[notFloran]

@nicolargo some informations about the bug :

• file: filter.py
• function : _is_process_filtered
• process: {'status': 'running', 'username': 'root', 'cpu_times': None, 'name': 'launchd', 'num_threads': None, 'cpu_percent': None, 'pid': 1, 'cmdline': None, 'memory_percent': None, 'memory_info': None, 'ppid': 0, 'gids': puids(real=0, effective=0, saved=0), 'nice': 0}
• key: cmdline

[nicolargo]

@notFloran Last commit on the DEVELOP branch should correct this issue coming from the fact that 'cmdline' is none...

[notFloran]

@nicolargo Another issue when I try to filter:

 File "/Users/floranbrutel/dev/glances/glances/plugins/glances_processlist.py", line 536, in __sum_stats
ret += p[key] TypeError: unsupported operand type(s) for +=: 'float' and 'NoneType'

• key: cpu_percent
• p: {'status': 'R', 'username': 'root', 'cpu_times': None, 'name': 'launchd', 'num_threads': None, 'cpu_percent': None, 'pid': 1, 'time_since_update': 0.21837711334228516, 'io_counters': [0, 0, 0, 0, 0], 'cmdline': None, 'memory_percent': None, 'memory_info': None, 'ppid': 0, 'gids': puids(real=0, effective=0, saved=0), 'nice': 0}

[nicolargo]

Step by step, bug by bug...

It should be corrected, can you test it ?

Note: it is very strange because your cmdline, cpu_percent and memory_percent are None... Perhaps, it came from another issue...

[notFloran]

The filters are now ok!

But I have an another issue with the column "TIME+" : when I launch glances the column has data and then after 2-3 seconds all data are replaced with "?". The log: https://pastebin.com/9A1y7LBf

[nicolargo]

For all processes ???

[notFloran]

Yes

[nicolargo]

Patched... Can you test (again ;)) ?

[notFloran]

It's all good !