[issue29168] multiprocessing pickle error

[Vinay Sajip]

Vinay Sajip added the comment:

> Have a look at the sample.py program.

It fails on Python 3.4.3 on my Linux system (just renamed to mp_sample.py):

python3 mp_sample.py 
starting (without multiprocessing pool)...
worker process 0
worker process 3
worker process 4
worker process 6
worker process 5
worker process 9
worker process 8
worker process 7
worker process 1
worker process 2
starting (with multiprocessing pool)...
Traceback (most recent call last):
  File "mp_sample.py", line 82, in <module>
    op.start()
  File "mp_sample.py", line 43, in start
    result.get()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 383, in _handle_tasks
    put(task)
  File "/usr/lib/python3.4/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/usr/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
  File "/usr/lib/python3.4/multiprocessing/queues.py", line 57, in __getstate__
    context.assert_spawning(self)
  File "/usr/lib/python3.4/multiprocessing/context.py", line 347, in assert_spawning
    ' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance

Also:

QueueHandler isn't designed to be pickleable, nor is that necessary for use with multiprocessing. The logging cookbook contains a working example with QueueHandler and multiple processes using multiprocessing, and I've just run it on Python 3.6 (and 3.7) with no problems. See

https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes

No idea why the sample program isn't working, and I'm sorry I haven't the time to look at it in detail. Please confirm if the linked sample works on your system, and if it does, see if you can adapt it to your needs. Note that it doesn't pickle any handlers.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue29168>
_______________________________________