Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

virtual_memory() gives highly inaccurate results #685

Closed
coderforlife opened this issue Sep 22, 2015 · 19 comments
Closed

virtual_memory() gives highly inaccurate results #685

coderforlife opened this issue Sep 22, 2015 · 19 comments
Labels
bug linux

Comments

@coderforlife
Copy link

These commands were done right after each other:

In [1]: import psutil

In [2]: psutil.virtual_memory()
Out[2]: svmem(total=50468696064L, available=4217393152L, percent=91.6, used=50133450752L, free=335245312L, active=2109054976, inactive=2236395520, buffers=372736L, cached=3881775104)

In [3]: !free
              total        used        free      shared  buff/cache   available
Mem:       49285836     1006960      327452       33552    47951424    47819748
Swap:      16777212       21152    16756060

free reports KiB and psutil gives bytes. After considering that, total is spot on and free is very close. The others are not though:

  • buff/cache is 8% of what it should be
  • used is 48x what is should be
  • available is 8% of what is should be

Overall it makes it look like I have only 4 GiB free when I should have ~46GB free.

Machine info:

  • Python v2.7.5
  • psutil v3.2.1 (latest on PIP)
  • CentOS Linux release 7.1.1503 (Core)
  • Linux Kernel 3.10.0-229.7.2.el7.x86_64 SMP
  • RAM: 47.00 GB
  • CPU(s): 24x Intel(R) Xeon(R) CPU X5690 @ 3.47GHz
@coderforlife
Copy link
Author

Same results with psutil v3.1.1.

@jchien14 jchien14 mentioned this issue Nov 17, 2015
Closed
@giampaolo
Copy link
Owner

From psutil doc:

free: memory not being used at all (zeroed) that is readily available; note that this doesn’t reflect the actual memory available (use ‘available’ instead).

In summary, in order to see the memory available you should rely on available, not free.
That would explain the difference with free cmdline utility.
Also, psutil.virtual_memory().available matches what shown by htop.

buff/cache is 8% of what it should be

I am not sure why this is different but both those values are tested against vmstat cmdline utility here:

psutil/test/_linux.py

Line 159 in c4c70ef

def test_vmem_buffers(self):

...and here:

psutil/test/_linux.py

Line 165 in c4c70ef

def test_vmem_cached(self):

Could you run the test suite (make test) and see if they fail?

used is 48x what is should be

This is likely because we are comparing oranges and apples. psutil determines used memory as in used = total - free. free cmdline utility probably does that differently.

@coderforlife
Copy link
Author

Sorry for the delayed response, but I have completed running tests. 3 tests fail: test_fetch_all, test_swapmem_free, and test_vmem_cached. The first one probably fails because of the network logins (it specifically reports it fails looking up UIDs and usernames). The other ones are more interesting and say the following:

======================================================================
FAIL: test_swapmem_free (_linux.LinuxSpecificTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/j1bush/temp/psutil/test/test_psutil.py", line 451, in wrapper
    return fun(*args, **kwargs)
  File "/home/j1bush/temp/psutil/test/_linux.py", line 188, in test_swapmem_free
    delta=MEMORY_TOLERANCE)
AssertionError: 17179865088 != 14361993216L within 512000 delta

======================================================================
FAIL: test_vmem_cached (_linux.LinuxSpecificTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/j1bush/temp/psutil/test/test_psutil.py", line 451, in wrapper
    return fun(*args, **kwargs)
  File "/home/j1bush/temp/psutil/test/_linux.py", line 169, in test_vmem_cached
    delta=MEMORY_TOLERANCE)
AssertionError: 189739008 != 38711296 within 512000 delta

----------------------------------------------------------------------

So swapmem is off by ~2.6 GiB and vmem off by about ~144 MiB (on a machine with 24 GiB of RAM).

On a different machine with a very similar setup but with 377.67 GiB of RAM I get just vmem_cached being off: AssertionError: 143214313472 != 92818538496 within 512000 delta which is ~47 GiB of RAM different.

(note: 59 tests were sipped, mostly due to lack of su priv)

The tests were run with 4582e6b (most recent on master).

giampaolo added a commit that referenced this issue Feb 15, 2016
@giampaolo
#685 / linux / mem funcs: return original C types and use the multipl…
50fd31a 
…ier in python in order to avoid C typing issues
@giampaolo
Copy link
Owner

Sorry for getting to this so late. This looks like a C typing issue. Can you please try revision 50fd31a and re-run tests?

@giampaolo
Copy link
Owner

OK, I consider this is fixed.

@coderforlife
Copy link
Author

Sorry for the massive delay in responding to this. I just checked out master again and ran make test and it still didn't work. The relevant failures are:

======================================================================
FAIL: test_cached (test_linux.TestSystemVirtualMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/j1bush/temp/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 397795065856 != 367770619904 within 512000 delta

======================================================================
FAIL: test_used (test_linux.TestSystemVirtualMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/j1bush/temp/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 5982253056 != 403786346496 within 512000 delta

(on the machine with 377.67 GiB of RAM). I did have to remove the --user from the python setup.py develop --user in the Makefile to make it run (it said --user wasn't a valid option).

There were additional failures as well (test_net_if_stats and test_net_io_counters both with a list index out of range, test_connections_unix with 5 != 7, test_cpu_times_comparison with 70147192.4 != 70147192.25999999 [just outside tolerance], and test_fetch_all like before).

So I guess this problem isn't really fixed. Once again sorry for the delay in checking.

@sacreman
Copy link

Hi, I am also having a similar issue on Centos 7 with 1024MB ram and I'm testing with psutil release-4.1.0.

However, my problem relates to calculating memory % used which seems to differ greatly from the output of the free command. A basic calculation on this machine with free shows 11% memory usage whereas with psutil it shows 22%. No matter what I try I can't seem to derive the 11% value from the psutil numbers.

[root@centos7 psutil]# uname -a
Linux centos7 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

The percentage calculation for memory used always seems to be very high compared to free -m.

Example output from free -m:

[root@centos7 bin]# free -m
              total        used        free      shared  buff/cache   available
Mem:            987         104         200          31         682         672
Swap:             0           0           0

The important thing to note here is the used being returned as 104mb.

To work out % used with free I have been doing:

(used / total ) * 100%
(109 / 987) * 100 = 11.0%

From the man page for free it says used is total - free - buffers - cache / total * 100%

The raw output I get from psutil is:

>>> import psutil
>>> psutil.virtual_memory()
svmem(total=1035649024, available=801570816, percent=22.6, used=833904640, free=201744384, active=390012928, inactive=305410048, buffers=82182144, cached=517644288)

Which in mb's is:

total = 1035.65mb
available = 801.57mb
used = 833.90mb
free = 201.74mb
active = 390.01mb
inactive = 305.41mb
buffers = 82.18mb
cached = 517.64mb

Here used is shown as 833mb which is quite high. So trying to derive the used number from psutil via the same method in free gives:

used = total - free - buffers - cache / total * 100%
used = (1011376 - 195480 - 81432 - 507888) = 226.57mb

It's still a lot more than the 104mb given by free -m. I believe this is the reason for the 11% usage from free -m vs 22% from psutil.

I've cloned the latest master to the same machine. Will paste the output of make test into the next comment.

BTW, thanks for psutil. It is an awesome library, I apologise in advance if I'm doing something silly with the numbers. My maths is not brilliant :)

@sacreman
Copy link

Output from make test on the same machine. I was going to cut only the failed memory tests out but the output isn't too long and I thought perhaps not seeing it all may hide some useful info.

[root@centos7 psutil]# make test
rm -f `find . -type f -name \*.py[co]`
rm -f `find . -type f -name \*.so`
rm -f `find . -type f -name \*.~`
rm -f `find . -type f -name \*.orig`
rm -f `find . -type f -name \*.bak`
rm -f `find . -type f -name \*.rej`
rm -rf `find . -type d -name __pycache__`
rm -rf *.core
rm -rf *.egg-info
rm -rf *\estfile*
rm -rf .coverage
rm -rf .tox
rm -rf build
rm -rf dist
rm -rf docs/_build
rm -rf htmlcov
python setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/psutil
copying psutil/_pswindows.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_common.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_pslinux.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_psosx.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_compat.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_psbsd.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_psposix.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/_pssunos.py -> build/lib.linux-x86_64-2.7/psutil
copying psutil/__init__.py -> build/lib.linux-x86_64-2.7/psutil
creating build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_osx.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_windows.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_process.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_posix.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_system.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_linux.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_memory_leaks.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/__init__.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_bsd.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/runner.py -> build/lib.linux-x86_64-2.7/psutil/tests
copying psutil/tests/test_sunos.py -> build/lib.linux-x86_64-2.7/psutil/tests
running build_ext
building 'psutil._psutil_linux' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/psutil
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DPSUTIL_VERSION=410 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
building 'psutil._psutil_posix' extension
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
gcc -pthread -shared -Wl,-z,relro build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -L/usr/lib64 -lpython2.7 -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
python setup.py build_ext -i
running build_ext
copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
python setup.py develop --user
running develop
running egg_info
creating psutil.egg-info
writing psutil.egg-info/PKG-INFO
writing top-level names to psutil.egg-info/top_level.txt
writing dependency_links to psutil.egg-info/dependency_links.txt
writing manifest file 'psutil.egg-info/SOURCES.txt'
reading manifest file 'psutil.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory 'docs/_build'
writing manifest file 'psutil.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
Creating /root/.local/lib/python2.7/site-packages/psutil.egg-link (link to .)
psutil 4.1.0 is already the active version in easy-install.pth

Installed /root/psutil
Processing dependencies for psutil==4.1.0
Finished processing dependencies for psutil==4.1.0
python psutil/tests/runner.py
test_process_create_time (test_osx.TestProcess) ... skipped 'not an OSX system'
test_cpu_count_logical (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_cpu_count_physical (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_disks (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_swapmem_sin (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_swapmem_sout (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_swapmem_total (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_vmem_active (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_vmem_free (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_vmem_inactive (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_vmem_total (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_vmem_wired (test_osx.TestSystemAPIs) ... skipped 'not an OSX system'
test_cmdline_32 (test_windows.RemoteProcessTestCase) ... skipped 'not a Windows system'
test_cmdline_64 (test_windows.RemoteProcessTestCase) ... skipped 'not a Windows system'
test_cwd_32 (test_windows.RemoteProcessTestCase) ... skipped 'not a Windows system'
test_cwd_64 (test_windows.RemoteProcessTestCase) ... skipped 'not a Windows system'
test_environ_32 (test_windows.RemoteProcessTestCase) ... skipped 'not a Windows system'
test_environ_64 (test_windows.RemoteProcessTestCase) ... skipped 'not a Windows system'
test_compare_name_exe (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_compare_values (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_cpu_times (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_create_time (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_io_counters (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_memory_info (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_name (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_num_handles (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_zombies (test_windows.TestDualProcessImplementation) ... skipped 'not a Windows system'
test_cpu_count (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_ctrl_signals (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_disks (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_exe (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_issue_24 (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_name_always_available (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_net_if_stats (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_nic_names (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_num_handles (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_num_handles_2 (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_pids (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_process_cmdline (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_process_create_time (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_process_exe (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_process_name (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_process_rss_memory (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_process_username (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_process_vms_memory (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_send_signal (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_special_pid (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_total_phymem (test_windows.WindowsSpecificTestCase) ... skipped 'not a Windows system'
test_Popen (test_process.LimitedUserTestCase) ... ok
test_as_dict (test_process.LimitedUserTestCase) ... ok
test_children (test_process.LimitedUserTestCase) ... ok
test_children_duplicates (test_process.LimitedUserTestCase) ... ok
test_children_recursive (test_process.LimitedUserTestCase) ... ok
test_cmdline (test_process.LimitedUserTestCase) ... ok
test_connection_constants (test_process.LimitedUserTestCase) ... ok
test_connection_fromfd (test_process.LimitedUserTestCase) ... ok
test_connections (test_process.LimitedUserTestCase) ... ok
test_connections_unix (test_process.LimitedUserTestCase) ... ok
test_cpu_affinity (test_process.LimitedUserTestCase) ... ok
test_cpu_percent (test_process.LimitedUserTestCase) ... ok
test_cpu_times (test_process.LimitedUserTestCase) ... ok
test_cpu_times_2 (test_process.LimitedUserTestCase) ... ok
test_create_time (test_process.LimitedUserTestCase) ... ok
test_cwd (test_process.LimitedUserTestCase) ... ok
test_cwd_2 (test_process.LimitedUserTestCase) ... ok
test_environ (test_process.LimitedUserTestCase) ... ok
test_exe (test_process.LimitedUserTestCase) ... ok
test_gids (test_process.LimitedUserTestCase) ... ok
test_halfway_terminated_process (test_process.LimitedUserTestCase) ... ok
test_invalid_pid (test_process.LimitedUserTestCase) ... ok
test_io_counters (test_process.LimitedUserTestCase) ... ok
test_ionice (test_process.LimitedUserTestCase) ... ok
test_is_running (test_process.LimitedUserTestCase) ... ok
test_kill (test_process.LimitedUserTestCase) ... ok
test_memory_full_info (test_process.LimitedUserTestCase) ... ok
test_memory_info (test_process.LimitedUserTestCase) ... ok
test_memory_maps (test_process.LimitedUserTestCase) ... ok
test_memory_percent (test_process.LimitedUserTestCase) ... ok
test_name (test_process.LimitedUserTestCase) ... ok
test_nice (test_process.LimitedUserTestCase) ... ok
test_num_ctx_switches (test_process.LimitedUserTestCase) ... ok
test_num_fds (test_process.LimitedUserTestCase) ... ok
test_num_handles (test_process.LimitedUserTestCase) ... ok
test_num_threads (test_process.LimitedUserTestCase) ... ok
test_open_files (test_process.LimitedUserTestCase) ... ok
test_open_files_2 (test_process.LimitedUserTestCase) ... ok
test_parent_ppid (test_process.LimitedUserTestCase) ... ok
test_pid (test_process.LimitedUserTestCase) ... ok
test_pid_0 (test_process.LimitedUserTestCase) ... ok
test_prog_w_funky_name (test_process.LimitedUserTestCase) ... ok
test_rlimit (test_process.LimitedUserTestCase) ... ok
test_rlimit_get (test_process.LimitedUserTestCase) ... ok
test_rlimit_infinity (test_process.LimitedUserTestCase) ... ok
test_rlimit_infinity_value (test_process.LimitedUserTestCase) ... ok
test_rlimit_set (test_process.LimitedUserTestCase) ... ok
test_send_signal (test_process.LimitedUserTestCase) ... ok
test_status (test_process.LimitedUserTestCase) ... ok
test_suspend_resume (test_process.LimitedUserTestCase) ... ok
test_terminal (test_process.LimitedUserTestCase) ... ok
test_terminate (test_process.LimitedUserTestCase) ... ok
test_threads (test_process.LimitedUserTestCase) ... ok
test_threads_2 (test_process.LimitedUserTestCase) ... ok
test_uids (test_process.LimitedUserTestCase) ... ok
test_username (test_process.LimitedUserTestCase) ... ok
test_wait (test_process.LimitedUserTestCase) ... ok
test_wait_non_children (test_process.LimitedUserTestCase) ... ok
test_wait_timeout_0 (test_process.LimitedUserTestCase) ... ok
test_weird_environ (test_process.LimitedUserTestCase) ... ok
test_zombie_process (test_process.LimitedUserTestCase) ... ok
test_fetch_all (test_process.TestFetchAllProcesses) ... ok
test_disk_usage (test_process.TestNonUnicode) ... ok
test_proc_cmdline (test_process.TestNonUnicode) ... ok
test_proc_cwd (test_process.TestNonUnicode) ... ok
test_proc_environ (test_process.TestNonUnicode) ... ok
test_proc_exe (test_process.TestNonUnicode) ... ok
test_proc_name (test_process.TestNonUnicode) ... ok
test_proc_open_files (test_process.TestNonUnicode) ... ok
test_Popen (test_process.TestProcess) ... ok
test_as_dict (test_process.TestProcess) ... ok
test_children (test_process.TestProcess) ... ok
test_children_duplicates (test_process.TestProcess) ... ok
test_children_recursive (test_process.TestProcess) ... ok
test_cmdline (test_process.TestProcess) ... ok
test_connection_constants (test_process.TestProcess) ... ok
test_connection_fromfd (test_process.TestProcess) ... ok
test_connections (test_process.TestProcess) ... ok
test_connections_unix (test_process.TestProcess) ... ok
test_cpu_affinity (test_process.TestProcess) ... ok
test_cpu_percent (test_process.TestProcess) ... ok
test_cpu_times (test_process.TestProcess) ... ok
test_cpu_times_2 (test_process.TestProcess) ... ok
test_create_time (test_process.TestProcess) ... ok
test_cwd (test_process.TestProcess) ... ok
test_cwd_2 (test_process.TestProcess) ... ok
test_environ (test_process.TestProcess) ... ok
test_exe (test_process.TestProcess) ... ok
test_gids (test_process.TestProcess) ... ok
test_halfway_terminated_process (test_process.TestProcess) ... ok
test_invalid_pid (test_process.TestProcess) ... ok
test_io_counters (test_process.TestProcess) ... ok
test_ionice (test_process.TestProcess) ... ok
test_is_running (test_process.TestProcess) ... ok
test_kill (test_process.TestProcess) ... ok
test_memory_full_info (test_process.TestProcess) ... ok
test_memory_info (test_process.TestProcess) ... ok
test_memory_maps (test_process.TestProcess) ... ok
test_memory_percent (test_process.TestProcess) ... ok
test_name (test_process.TestProcess) ... ok
test_nice (test_process.TestProcess) ... ok
test_num_ctx_switches (test_process.TestProcess) ... ok
test_num_fds (test_process.TestProcess) ... ok
test_num_handles (test_process.TestProcess) ... skipped 'Windows only'
test_num_threads (test_process.TestProcess) ... ok
test_open_files (test_process.TestProcess) ... ok
test_open_files_2 (test_process.TestProcess) ... ok
test_parent_ppid (test_process.TestProcess) ... ok
test_pid (test_process.TestProcess) ... ok
test_pid_0 (test_process.TestProcess) ... ok
test_prog_w_funky_name (test_process.TestProcess) ... ok
test_rlimit (test_process.TestProcess) ... ok
test_rlimit_get (test_process.TestProcess) ... ok
test_rlimit_infinity (test_process.TestProcess) ... ok
test_rlimit_infinity_value (test_process.TestProcess) ... ok
test_rlimit_set (test_process.TestProcess) ... ok
test_send_signal (test_process.TestProcess) ... ok
test_status (test_process.TestProcess) ... ok
test_suspend_resume (test_process.TestProcess) ... ok
test_terminal (test_process.TestProcess) ... ok
test_terminate (test_process.TestProcess) ... ok
test_threads (test_process.TestProcess) ... ok
test_threads_2 (test_process.TestProcess) ... ok
test_uids (test_process.TestProcess) ... ok
test_username (test_process.TestProcess) ... ok
test_wait (test_process.TestProcess) ... ok
test_wait_non_children (test_process.TestProcess) ... ok
test_wait_timeout_0 (test_process.TestProcess) ... ok
test_weird_environ (test_process.TestProcess) ... ok
test_zombie_process (test_process.TestProcess) ... ERROR
test_disk_usage (test_process.TestUnicode) ... ok
test_proc_cmdline (test_process.TestUnicode) ... ok
test_proc_cwd (test_process.TestUnicode) ... ok
test_proc_environ (test_process.TestUnicode) ... ok
test_proc_exe (test_process.TestUnicode) ... ok
test_proc_name (test_process.TestUnicode) ... ok
test_proc_open_files (test_process.TestUnicode) ... ok
test_cmdline (test_posix.TestProcess) ... ok
test_create_time (test_posix.TestProcess) ... ok
test_cwd (test_posix.TestProcess) ... ok
test_exe (test_posix.TestProcess) ... ok
test_gid (test_posix.TestProcess) ... ok
test_name (test_posix.TestProcess) ... ok
test_nice (test_posix.TestProcess) ... ok
test_num_fds (test_posix.TestProcess) ... ok
test_ppid (test_posix.TestProcess) ... ok
test_rss_memory (test_posix.TestProcess) ... ok
test_uid (test_posix.TestProcess) ... ok
test_username (test_posix.TestProcess) ... ok
test_vsz_memory (test_posix.TestProcess) ... ok
test_nic_names (test_posix.TestSystemAPIs) ... ok
test_os_waitpid_bad_ret_status (test_posix.TestSystemAPIs) ... ok
test_os_waitpid_eintr (test_posix.TestSystemAPIs) ... ok
test_os_waitpid_let_raise (test_posix.TestSystemAPIs) ... ok
test_pid_exists_let_raise (test_posix.TestSystemAPIs) ... ok
test_pids (test_posix.TestSystemAPIs) ... ok
test_users (test_posix.TestSystemAPIs) ... ok
test_PAGESIZE (test_system.TestSystemAPIs) ... ok
test_boot_time (test_system.TestSystemAPIs) ... ok
test_cpu_count (test_system.TestSystemAPIs) ... ok
test_cpu_percent (test_system.TestSystemAPIs) ... ok
test_cpu_stats (test_system.TestSystemAPIs) ... ok
test_cpu_times (test_system.TestSystemAPIs) ... ok
test_cpu_times_comparison (test_system.TestSystemAPIs) ... ok
test_cpu_times_percent (test_system.TestSystemAPIs) ... ok
test_cpu_times_time_increases (test_system.TestSystemAPIs) ... ok
test_disk_io_counters (test_system.TestSystemAPIs) ... ok
test_disk_partitions (test_system.TestSystemAPIs) ... ok
test_disk_usage (test_system.TestSystemAPIs) ... ok
test_disk_usage_unicode (test_system.TestSystemAPIs) ... ok
test_net_connections (test_system.TestSystemAPIs) ... ok
test_net_if_addrs (test_system.TestSystemAPIs) ... ok
test_net_if_addrs_mac_null_bytes (test_system.TestSystemAPIs) ... ok
test_net_if_stats (test_system.TestSystemAPIs) ... ok
test_net_io_counters (test_system.TestSystemAPIs) ... ok
test_per_cpu_percent (test_system.TestSystemAPIs) ... ok
test_per_cpu_times (test_system.TestSystemAPIs) ... ok
test_per_cpu_times_2 (test_system.TestSystemAPIs) ... ok
test_per_cpu_times_percent (test_system.TestSystemAPIs) ... ok
test_per_cpu_times_percent_negative (test_system.TestSystemAPIs) ... ok
test_pid_exists (test_system.TestSystemAPIs) ... ok
test_pid_exists_2 (test_system.TestSystemAPIs) ... ok
test_pids (test_system.TestSystemAPIs) ... ok
test_process_iter (test_system.TestSystemAPIs) ... ok
test_swap_memory (test_system.TestSystemAPIs) ... ok
test_test (test_system.TestSystemAPIs) ... ok
test_users (test_system.TestSystemAPIs) ... ok
test_virtual_memory (test_system.TestSystemAPIs) ... ok
test_wait_procs (test_system.TestSystemAPIs) ... ok
test_wait_procs_no_timeout (test_system.TestSystemAPIs) ... ok
test_boot_time_mocked (test_linux.TestMisc) ... ok
test_no_procfs_on_import (test_linux.TestMisc) ... ok
test_prlimit_availability (test_linux.TestMisc) ... ok
test_procfs_path (test_linux.TestMisc) ... ok
test_resource_consts_kernel_v (test_linux.TestMisc) ... ok
test_sector_size_mock (test_linux.TestMisc) ... ok
test_users_mocked (test_linux.TestMisc) ... ok
test_cmdline_mocked (test_linux.TestProcess) ... ok
test_exe_mocked (test_linux.TestProcess) ... ok
test_gids_mocked (test_linux.TestProcess) ... ok
test_io_counters_mocked (test_linux.TestProcess) ... ok
test_memory_full_info (test_linux.TestProcess) ... ok
test_memory_maps (test_linux.TestProcess) ... ok
test_num_ctx_switches_mocked (test_linux.TestProcess) ... ok
test_num_threads_mocked (test_linux.TestProcess) ... ok
test_open_files_file_gone (test_linux.TestProcess) ... ok
test_open_files_mode (test_linux.TestProcess) ... ok
test_ppid_mocked (test_linux.TestProcess) ... ok
test_readlink_path_deleted_mocked (test_linux.TestProcess) ... ok
test_terminal_mocked (test_linux.TestProcess) ... ok
test_threads_mocked (test_linux.TestProcess) ... ok
test_uids_mocked (test_linux.TestProcess) ... ok
test_cpu_count_logical_mocked (test_linux.TestSystemCPU) ... ok
test_cpu_count_logical_w_lscpu (test_linux.TestSystemCPU) ... ok
test_cpu_count_logical_w_nproc (test_linux.TestSystemCPU) ... ok
test_cpu_count_physical_mocked (test_linux.TestSystemCPU) ... ok
test_cpu_times (test_linux.TestSystemCPU) ... ok
test_disk_io_counters_kernel_2_4_mocked (test_linux.TestSystemDisks) ... ok
test_disk_io_counters_kernel_2_6_full_mocked (test_linux.TestSystemDisks) ... ok
test_disk_io_counters_kernel_2_6_limited_mocked (test_linux.TestSystemDisks) ... ok
test_disk_partitions_and_usage (test_linux.TestSystemDisks) ... ok
test_disk_partitions_mocked (test_linux.TestSystemDisks) ... ok
test_net_connections_ipv6_unsupported (test_linux.TestSystemNetwork) ... ok
test_net_connections_mocked (test_linux.TestSystemNetwork) ... ok
test_net_if_addrs_ips (test_linux.TestSystemNetwork) ... ok
test_net_if_names (test_linux.TestSystemNetwork) ... ok
test_net_if_stats (test_linux.TestSystemNetwork) ... ERROR
test_net_io_counters (test_linux.TestSystemNetwork) ... ERROR
test_free (test_linux.TestSystemSwapMemory) ... ok
test_no_vmstat_mocked (test_linux.TestSystemSwapMemory) ... ok
test_total (test_linux.TestSystemSwapMemory) ... ok
test_used (test_linux.TestSystemSwapMemory) ... ok
test_warnings_mocked (test_linux.TestSystemSwapMemory) ... ok
test_buffers (test_linux.TestSystemVirtualMemory) ... ok
test_cached (test_linux.TestSystemVirtualMemory) ... FAIL
test_free (test_linux.TestSystemVirtualMemory) ... ok
test_total (test_linux.TestSystemVirtualMemory) ... ok
test_used (test_linux.TestSystemVirtualMemory) ... FAIL
test_warnings_mocked (test_linux.TestSystemVirtualMemory) ... ok
test__all__ (test_misc.TestMisc) ... ok
test_access_denied__repr__ (test_misc.TestMisc) ... ok
test_ad_on_process_creation (test_misc.TestMisc) ... ok
test_isfile_strict (test_misc.TestMisc) ... ok
test_memoize (test_misc.TestMisc) ... ok
test_no_such_process__repr__ (test_misc.TestMisc) ... ok
test_parse_environ_block (test_misc.TestMisc) ... ok
test_process__eq__ (test_misc.TestMisc) ... ok
test_process__hash__ (test_misc.TestMisc) ... ok
test_process__repr__ (test_misc.TestMisc) ... ok
test_process__str__ (test_misc.TestMisc) ... ok
test_psutil_is_reloadable (test_misc.TestMisc) ... ok
test_sanity_version_check (test_misc.TestMisc) ... ok
test_serialization (test_misc.TestMisc) ... ok
test_setup_script (test_misc.TestMisc) ... ok
test_supports_ipv6 (test_misc.TestMisc) ... ok
test_timeout_expired__repr__ (test_misc.TestMisc) ... ok
test_version (test_misc.TestMisc) ... ok
test_zombie_process__repr__ (test_misc.TestMisc) ... ok
test_check_presence (test_misc.TestScripts) ... ok
test_disk_usage (test_misc.TestScripts) ... ok
test_executable (test_misc.TestScripts) ... ok
test_free (test_misc.TestScripts) ... ok
test_ifconfig (test_misc.TestScripts) ... ok
test_iotop (test_misc.TestScripts) ... ok
test_killall (test_misc.TestScripts) ... ok
test_meminfo (test_misc.TestScripts) ... ok
test_netstat (test_misc.TestScripts) ... ok
test_nettop (test_misc.TestScripts) ... ok
test_pidof (test_misc.TestScripts) ... ok
test_pmap (test_misc.TestScripts) ... ok
test_procinfo (test_misc.TestScripts) ... ok
test_procsmem (test_misc.TestScripts) ... ok
test_ps (test_misc.TestScripts) ... ok
test_pstree (test_misc.TestScripts) ... ok
test_top (test_misc.TestScripts) ... ok
test_who (test_misc.TestScripts) ... ok
test_cpu_count_logical (test_bsd.BSDSpecificTestCase) ... skipped 'not a BSD system'
test_disks (test_bsd.BSDSpecificTestCase) ... skipped 'not a BSD system'
test_process_create_time (test_bsd.BSDSpecificTestCase) ... skipped 'not a BSD system'
test_virtual_memory_total (test_bsd.BSDSpecificTestCase) ... skipped 'not a BSD system'
test_boot_time (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_cmdline (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_cpu_stats_ctx_switches (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_cpu_stats_interrupts (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_cpu_stats_soft_interrupts (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_cpu_stats_syscalls (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_exe (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_memory_maps (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_muse_vmem_active (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_muse_vmem_buffers (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_muse_vmem_cached (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_muse_vmem_free (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_muse_vmem_inactive (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_muse_vmem_total (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_muse_vmem_wired (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_uids_gids (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_vmem_active (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_vmem_buffers (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_vmem_cached (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_vmem_free (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_vmem_inactive (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_vmem_wired (test_bsd.FreeBSDSpecificTestCase) ... skipped 'not a FreeBSD system'
test_cpu_stats_ctx_switches (test_bsd.NetBSDSpecificTestCase) ... skipped 'not a NetBSD system'
test_cpu_stats_interrupts (test_bsd.NetBSDSpecificTestCase) ... skipped 'not a NetBSD system'
test_swapmem_free (test_bsd.NetBSDSpecificTestCase) ... skipped 'not a NetBSD system'
test_swapmem_total (test_bsd.NetBSDSpecificTestCase) ... skipped 'not a NetBSD system'
test_vmem_buffers (test_bsd.NetBSDSpecificTestCase) ... skipped 'not a NetBSD system'
test_vmem_shared (test_bsd.NetBSDSpecificTestCase) ... skipped 'not a NetBSD system'
test_boot_time (test_bsd.OpenBSDSpecificTestCase) ... skipped 'not an OpenBSD system'
test_swap_memory (test_sunos.SunOSSpecificTestCase) ... skipped 'not a SunOS system'

======================================================================
ERROR: test_zombie_process (test_process.TestProcess)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/psutil/psutil/tests/test_process.py", line 1342, in test_zombie_process
    zproc.as_dict()
  File "/root/psutil/psutil/__init__.py", line 474, in as_dict
    ret = attr()
  File "/root/psutil/psutil/__init__.py", line 656, in cwd
    return self._proc.cwd()
  File "/root/psutil/psutil/_pslinux.py", line 905, in wrapper
    raise NoSuchProcess(self.pid, self._name)
NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=29777)

======================================================================
ERROR: test_net_if_stats (test_linux.TestSystemNetwork)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/psutil/psutil/tests/test_linux.py", line 345, in test_net_if_stats
    int(re.findall('MTU:(\d+)', out)[0]))
IndexError: list index out of range

======================================================================
ERROR: test_net_io_counters (test_linux.TestSystemNetwork)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/psutil/psutil/tests/test_linux.py", line 363, in test_net_io_counters
    ifconfig_ret = ifconfig(name)
  File "/root/psutil/psutil/tests/test_linux.py", line 351, in ifconfig
    ret['packets_recv'] = int(re.findall('RX packets:(\d+)', out)[0])
IndexError: list index out of range

======================================================================
FAIL: test_cached (test_linux.TestSystemVirtualMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 690601984 != 552660992 within 512000 delta

======================================================================
FAIL: test_used (test_linux.TestSystemVirtualMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 157712384 != 953479168 within 512000 delta

----------------------------------------------------------------------
Ran 358 tests in 6.249s

FAILED (failures=2, errors=3, skipped=83)
make: *** [test] Error 1

@sacreman
Copy link

I'm guessing the test_used test is probably the problem.

======================================================================
FAIL: test_used (test_linux.TestSystemVirtualMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 157712384 != 953479168 within 512000 delta

157mb vs 953mb being returned. Which is closer to the 104mb being returned when running free -m plus the memory overhead of running the tests etc.

@giampaolo giampaolo reopened this Mar 29, 2016
@giampaolo
Copy link
Owner

used

You state that free calculates used memory as used = total - free - buffers - cache / total * 100% whereas psutil just do used = total - free.
Note that the resulting value matches my version of free (Ubuntu 14.04, kernel 3.13) which does not state that used is calculated like that. I tried to change the math and do used = total - free - buffers - cache instead and it broke my test.
This probably means that free changed the math at some point and honestly I'm not sure which one is correct (also I don't remember how I came up with used = total - free but I believe I copied it from zabbix).

cached

This is also weird. cached mem comes directly from /proc/meminfo (line starting with "Cached:"). If free result is different then I'm also tempted to think that it retrieves it in another (erroneous) way.

In summary, to me it appears that a certain version of "free" is wrong, not psutil. In this case perhaps it's best to "trust" the latest free version? What version of free do you have (try "free -V")? Mine is 3.3.9.

@giampaolo giampaolo mentioned this issue Mar 29, 2016
Closed
@giampaolo
Copy link
Owner

Note: in #800 I added "shared" memory. You may want to use that to play with numbers a little bit and see if you can match free numbers somehow.

@coderforlife
Copy link
Author

Just checked out the latest to give you accurate results.

First, the free version is free from procps-ng 3.3.10.

The two memory-based failing tests outputs are (on a machine with ~377 GiB of RAM):

======================================================================
FAIL: test_cached (test_linux.TestSystemVirtualMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/j1bush/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 393174581248 != 199509286912 within 512000 delta

======================================================================
FAIL: test_used (test_linux.TestSystemVirtualMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/j1bush/psutil/psutil/tests/__init__.py", line 494, in wrapper
    raise err
AssertionError: 5185323008 != 398376923136 within 512000 delta

----------------------------------------------------------------------

For the caches amount (which is off by about 190 GiB), grep Cached /proc/meminfo shows me the right-hand number. free and top report the left-hand number (but call it buff/cache). Although, from /proc/meminfo, Buffers is only about 16 MiB, so that doesn't add up.

@coderforlife
Copy link
Author

So I read the manpage for my version of free, and it says cached is coming from the Cached and Slab lines of /proc/meminfo. Looking at meminfo, slab is ~180GiB which then gets the numbers pretty close. I don't know what slab memory really is, or if it should be counted, but that is an explanation.

If it is not included in the cached value, then it should be given as a separate value, and adding it into used is then a decision that still needs to be made.

@coderforlife
Copy link
Author

And a discussion about free and slab memory is available at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=565518

@coderforlife
Copy link
Author

And a description of the values Slab, SReclaimable, and SUnreclaim: https://access.redhat.com/solutions/406773

It seems that SReclaimable is that memory that could easily be used by other processes if necessary while SUnreclaim cannot be. SReclaimable + SUnreclaim = Slab.

On this particular machine with ~377.7 GiB of RAM, SUnreclaim is 170.5 MiB while SReclaimable is 180.5 GiB.

So in the end, I believe that if SReclaimable is in /proc/meminfo, it needs to be added to cached. If only Slab (but not SReclaimable or SUnreclaim) it should probably be added to cached as well. This will allow the used data to be much MUCH more useful, as on high-memory machines this 50% chunk of total memory is quite a lot...

@sacreman
Copy link

This Centos 7 machine is also on 3.3.10:

[root@centos7 ~]# free -V
free from procps-ng 3.3.10

The first page of man free is:

FREE(1)                                                                  User Commands                                                                  FREE(1)

NAME
       free - Display amount of free and used memory in the system

SYNOPSIS
       free [options]

DESCRIPTION
       free displays the total amount of free and used physical and swap memory in the system, as well as the buffers and caches used by the kernel. The infor‐
       mation is gathered by parsing /proc/meminfo. The displayed columns are:

       total  Total installed memory (MemTotal and SwapTotal in /proc/meminfo)

       used   Used memory (calculated as total - free - buffers - cache)

       free   Unused memory (MemFree and SwapFree in /proc/meminfo)

       shared Memory used (mostly) by tmpfs (Shmem in /proc/meminfo, available on kernels 2.6.32, displayed as zero if not available)

       buffers
              Memory used by kernel buffers (Buffers in /proc/meminfo)

       cache  Memory used by the page cache and slabs (Cached and Slab in /proc/meminfo)

Will give it at test on one of our ubuntu 12.04 boxes. It is very possible that the Centos 7 free package is doing something funky.

@sacreman
Copy link

On an Ubuntu 12.04 the test passes for test_used. And the numbers from free are:

~# free -m
             total       used       free     shared    buffers     cached
Mem:         64457      33593      30863          0        181      15594
-/+ buffers/cache:      17817      46639
Swap:         1021          0       1021

Compared to the output of psutil:

>>> import psutil
>>> psutil.virtual_memory()
svmem(total=67588292608, available=48907816960, percent=27.6, used=35269840896, free=32318451712, active=26230091776, inactive=7874027520, buffers=190222336, cached=16399142912)
>>>

The discrepancy does appear to be with Centos7 with the procps-ng package.

I had a quick browse around the procps-ng gitlab repo and couldn't really work out if this was a brand new package that differed to the one on Ubuntu.

@giampaolo giampaolo mentioned this issue Sep 10, 2016
Closed
@Deewiant
Copy link

Just noting that SReclaimable is indeed added to the "cache" value reported by free as seen at https://gitlab.com/procps-ng/procps/blob/24fd2605c51fccc375ab0287cec33aa767f06718/proc/sysinfo.c#L762 (tip of the master branch at time of writing this comment), so it'd make sense to do the same thing in psutil's virtual_memory().

@giampaolo giampaolo mentioned this issue Sep 18, 2016
Closed
@giampaolo
Copy link
Owner

Fixed in #887

@walking-octopus walking-octopus mentioned this issue Sep 10, 2022
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug linux
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Deewiant @giampaolo @coderforlife @sacreman