multiprocessing Pool workers cannot create subprocesses

Jason Grout jason-python at creativetrax.com
Fri Mar 18 12:49:44 EDT 2011


In a recent application, a student of mine tried to create child 
processes inside of a multiprocessing Pool worker (for security and 
convenience reasons, we wanted to run some code inside of a child 
process).  Here is some test code for python 2.7:

=============================
import multiprocessing as mp

def run_computation(x):
     print 2*x

def f(x):
     curr_proc=mp.current_process()
     # uncomment following line to get this to work
     #curr_proc.daemon=False

     p = mp.Process(target=run_computation, args=(x,))
     p.start()
     p.join()


pool = mp.Pool(processes=4)
pool.map(f, range(10))

===============================

The result is:

Traceback (most recent call last):
   File "daemon.py", line 17, in <module>
     pool.map(f, range(10))
   File 
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", 
line 199, in map
     return self.map_async(func, iterable, chunksize).get()
   File 
"/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", 
line 491, in get
     raise self._value
AssertionError: daemonic processes are not allowed to have children

The problem appears to be that multiprocessing sets its workers to have 
the daemon flag set to True, which prevents workers from creating child 
processes.  If I uncomment the line indicated in the code, I can create 
child processes and the program works (it prints out the even numbers 
from 0 to 18).

It makes me nervous to just change the daemon status of the process like 
that, especially when I don't know the reason the workers have 
daemon=True to begin with.  What is the reasoning behind that decision? 
  What issues do we need to worry about if we just set the daemon mode 
flag like in the above code?

Thanks,

Jason



More information about the Python-list mailing list