Process.open_files fails if deleted files still visible

[skundrik]

os.readlink(file) can sometimes return path looking like this (^@ is NULL)

/home/testaccount/.logstash-forwarder^@^@^@^@ (deleted)
/home/testaccount/.logstash-forwarder^@new (deleted)

resulting in

TypeError: must be encoded string without NULL bytes, not str

when call to isfile_strict(file) is made. It seems to be happening every time I try to use open_files(proc).

Environment
OS: 2.6.18-164.10.1.el5 #1 SMP Wed Dec 30 18:35:28 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
Python 2.7.10 |Anaconda 2.3.0 (64-bit)| (default, May 28 2015, 17:02:03)
psutil 3.3.0

[giampaolo]

Can you paste the original traceback message?

[skundrik]

Here you go

Traceback (most recent call last):
  File "/home/testaccount/testScript.py", line 97, in <module>
    openFiles = proc.open_files()
  File "/home/testaccount/Anaconda/current_Anaconda_dev_64/lib/python2.7/site-packages/psutil/__init__.py", line 962, in open_files
    return self._proc.open_files()
  File "/home/testaccount/Anaconda/current_Anaconda_dev_64/lib/python2.7/site-packages/psutil/_pslinux.py", line 645, in wrapper
    return fun(self, *args, **kwargs)
  File "/home/testaccount/Anaconda/current_Anaconda_dev_64/lib/python2.7/site-packages/psutil/_pslinux.py", line 1052, in open_files
    if file.startswith('/') and isfile_strict(file):
  File "/home/testaccount/Anaconda/current_Anaconda_dev_64/lib/python2.7/site-packages/psutil/_common.py", line 154, in isfile_strict
    st = os.stat(path)
TypeError: must be encoded string without NULL bytes, not str

[giampaolo]

Can you try doing isfile_strict(os.path.normpath(file)) and see if it fixes the issue?

[skundrik]

Nope, doesn't seem to fix it as it doesn't remove the NULL characters.

[giampaolo]

Can you put a pdb in there and copy & paste the actual string?

[skundrik]

Traceback (most recent call last):
  File "/home/testaccount/Anaconda/Anaconda-2.2.0-Linux-x86_64/envs/anaconda_2_3_0/lib/python2.7/pdb.py", line 1314, in main
    pdb._runscript(mainpyfile)
  File "/home/testaccount/Anaconda/Anaconda-2.2.0-Linux-x86_64/envs/anaconda_2_3_0/lib/python2.7/pdb.py", line 1233, in _runscript
    self.run(statement)
  File "/home/testaccount/Anaconda/Anaconda-2.2.0-Linux-x86_64/envs/anaconda_2_3_0/lib/python2.7/bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "/home/testaccount/runLogstashForwarder.py", line 1, in <module>
    import psutil
  File "/home/testaccount/Anaconda/Anaconda-2.2.0-Linux-x86_64/envs/anaconda_2_3_0/lib/python2.7/site-packages/psutil/__init__.py", line 962, in open_files
    return self._proc.open_files()
  File "/home/testaccount/Anaconda/Anaconda-2.2.0-Linux-x86_64/envs/anaconda_2_3_0/lib/python2.7/site-packages/psutil/_pslinux.py", line 645, in wrapper
    return fun(self, *args, **kwargs)
  File "/home/testaccount/Anaconda/Anaconda-2.2.0-Linux-x86_64/envs/anaconda_2_3_0/lib/python2.7/site-packages/psutil/_pslinux.py", line 1052, in open_files
    if file.startswith('/') and isfile_strict(file):
  File "/home/testaccount/Anaconda/Anaconda-2.2.0-Linux-x86_64/envs/anaconda_2_3_0/lib/python2.7/site-packages/psutil/_common.py", line 154, in isfile_strict
    st = os.stat(path)
TypeError: must be encoded string without NULL bytes, not str
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/testaccount/Anaconda/Anaconda-2.2.0-Linux-x86_64/envs/anaconda_2_3_0/lib/python2.7/site-packages/psutil/_common.py(154)isfile_strict()
-> st = os.stat(path)
(Pdb) p path
'/home/testhome/.logstash-forwarder\x00\x00\x00\x00 (deleted)'
(Pdb)

[giampaolo]

I should have fixed this (29334f1).

[skundrik]

What about when the path is '/home/testhome/.logstash-forwarder\x00new (deleted)'? Maybe a better way would be to strip everything starting with first \x00 character. It looks to me like os.readlink() is copying more into the return string than it should.

[giampaolo]

What about when the path is '/home/testhome/.logstash-forwarder\x00new (deleted)'?

Apparently any string containing \x00 will result in a TypeError if passed to os.* functions so it seems the right way to go is to remove all null bytes.

[skundrik]

I was referring to this part of the readlink() function in _pslinux.py

+    # Certain paths have ' (deleted)' appended. Usually this is
+    # bogus as the file actually exists.
+    if path.endswith(' (deleted)') and not path_exists_strict(path):
+        path = path[:-10]

[giampaolo]

What about it?

[skundrik]

It only strips ' (deleted)' but the path can also end with 'new (deleted)'. Shouldn't that be stripped as well? Or everything after the first '\x00' is encountered?

[giampaolo]

Uhm... I've just verified that there can be paths with the (deleted) part appended but without any null byte, e.g.: /run/shm/.com.google.Chrome.58jAm8 (deleted). I think you are right though: we should split('\x00')[0] and discard everything after the null byte. Done in 9e3453c.

[skundrik]

👍 Thank you.