[pypy-issue] Issue #3154: Multiprocessing leaking file descriptors (pypy/pypy)
Andrew Robinson
issues-reply at bitbucket.org
Thu Jan 16 07:20:54 EST 2020
New issue 3154: Multiprocessing leaking file descriptors
https://bitbucket.org/pypy/pypy/issues/3154/multiprocessing-leaking-file-descriptors
Andrew Robinson:
Have migrated a multiprocessing project from PyPy 6.0 \(2.7\) to PyPy 7.3 \(3.6\) and have identified an issue with multiprocessing leaking descriptors. I believe this may also relate to the issue reported as #3060.
I have identified this issue as repeatable on both Ubuntu 18.04.2 and CentOS 7.0.
Simple code to reproduce issue \(Taken from #3111 which is raised a different issue, but current issue is reproduced with the same code sample\):
```
import multiprocessing
def process_chunk(chunk_index):
return ([],[],[])
def main():
i = 0
with multiprocessing.Pool(5, maxtasksperchild=1) as p:
for sub_headers, sub_results, sub_scores in p.imap_unordered(process_chunk, range(5000)):
print(i)
i += 1
if __name__ == '__main__':
main()
```
If saving the above as “test.py” the descriptor leak can be verified by running “/path/to/pypy3 test.py” then taking the PID from ps and doing an “ls -l /proc/PID/fd” and comparing to the same when running native Python Ie. Running “/path/to/python3 test.py” taking PID and “ls -l /proc/PID/fd”.
Under native Python3 the descriptor list remains constant at 9 pipe entries.
Under PyPy the file descriptor list keeps growing until ulimit is reached and it will throw an exception as “OSError: \[Errno 24\] Too many open files“. In my case ulimit is set to 1024.
Traceback below:
```
Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/callation/pypy3/lib-python/3/threading.py", line 916, in _bootstrap_inner
self.run()
File "/opt/callation/pypy3/lib-python/3/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 405, in _handle_workers
pool._maintain_pool()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 246, in _maintain_pool
self._repopulate_pool()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/pool.py", line 239, in _repopulate_pool
w.start()
File "/opt/callation/pypy3/lib-python/3/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/opt/callation/pypy3/lib-python/3/multiprocessing/popen_fork.py", line 65, in _launch
parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files
```
More information about the pypy-issue
mailing list