[pypy-issue] [issue1289] multiprocessing Pool + maxtasksperchild + signal = ValueError

[Alexey Kryuchkov]

New submission from Alexey Kryuchkov <alexey.kruchkov at gmail.com>:

The following program runs correctly under CPython 2.7, but produces errors under PyPy:

# ---- bad.py ---------------------------------------------------
from multiprocessing import Pool
from signal import signal, SIGINT, SIG_DFL

def worker_init():
    # actual parameters of 'signal' have no influence on the outcome
    signal(SIGINT, SIG_DFL)

def worker_body(arg):
    print 'processing %s' % arg

def main():
    # 1) note the 'maxtasksperchild' parameter. The idea is that
    #    worker processes restart after every processed task.
    # 2) actual number of processes has no influence on the outcome
    pool = Pool(processes=1, maxtasksperchild=1, initializer=worker_init)
    sequence = [1, 2, 3, 4]
    pool.map(worker_body, sequence, chunksize=1)

if __name__ == '__main__':
    main()
# ---- end of bad.py ----------------------------------------------------

Expected output (which is also actual output under CPython 2.7) is:

    ~$ python bad.py
    processing 1
    processing 2
    processing 3
    processing 4
    ~$

Actual output under PyPy 1.9 is:

    ~$ pypy bad.py
    processing 1
    Process PoolWorker-2:
    Traceback (most recent call last):
      File "/opt/pypy-1.9/lib-python/2.7/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/opt/pypy-1.9/lib-python/2.7/multiprocessing/process.py", line 114, in run
        self._target(*self._args, **self._kwargs)
      File "/opt/pypy-1.9/lib-python/2.7/multiprocessing/pool.py", line 80, in worker
        initializer(*initargs)
      File "badpypy.py", line 5, in worker_init
        signal(SIGINT, SIG_DFL)
    ValueError: signal() must be called from the main thread
    Process PoolWorker-3:
    Traceback (most recent call last):
      File "/opt/pypy-1.9/lib-python/2.7/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/opt/pypy-1.9/lib-python/2.7/multiprocessing/process.py", line 114, in run
        self._target(*self._args, **self._kwargs)
      File "/opt/pypy-1.9/lib-python/2.7/multiprocessing/pool.py", line 80, in worker
        initializer(*initargs)
      File "badpypy.py", line 5, in worker_init
        signal(SIGINT, SIG_DFL)
    ValueError: signal() must be called from the main thread
    Process PoolWorker-4:
    (...and so on, until the process is killed manually)

Same program without 'maxtasksperchild' parameter executes correctly under PyPy:

# ---- good.py --------------------------------------
from multiprocessing import Pool
from signal import signal, SIGINT, SIG_DFL

def worker_init():
    signal(SIGINT, SIG_DFL)

def worker_body(arg):
    print 'processing %s' % arg

def main():
    # removed 'maxtasksperchild' parameter
    pool = Pool(processes=1, initializer=worker_init)
    sequence = [1, 2, 3, 4]
    pool.map(worker_body, sequence, chunksize=1)

if __name__ == '__main__':
    main()
# ---- end of good.py ---------------------------------

Its output under both PyPy and CPython is the same as CPython's output for bad.py.

My environment:

~$ pypy --version
Python 2.7.2 (341e1e3821ff, Jun 07 2012, 15:38:48)
[PyPy 1.9.0 with GCC 4.4.3]
~$ python --version
Python 2.7.1+
~$ uname -a
Linux ALEXU31S 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:17:25 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
~$ cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.04
DISTRIB_CODENAME=natty
DISTRIB_DESCRIPTION="Ubuntu 11.04"

----------
messages: 4849
nosy: babazka, pypy-issue
priority: bug
release: 1.9
status: unread
title: multiprocessing Pool + maxtasksperchild + signal = ValueError

________________________________________
PyPy bug tracker <tracker at bugs.pypy.org>
<https://bugs.pypy.org/issue1289>
________________________________________