[issue17560] problem using multiprocessing with really big objects?

[mrjbq7]

New submission from mrjbq7:

I ran into a problem using multiprocessing to create large data objects (in this case numpy float64 arrays with 90,000 columns and 5,000 rows) and return them to the original python process.

It breaks in both Python 2.7 and 3.3, using numpy 1.7.0 (but with different error messages).

It is possible the array is too large to be serialized (450 million 64-bit numbers exceeds a 32-bit limit)?


Python 2.7
==========

Process PoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 99, in worker
    put((job, i, result))
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 390, in put
    return send(obj)
SystemError: NULL result without error in PyObject_Call


Python 3.3
==========

Traceback (most recent call last):
  File "multi.py", line 18, in <module>
    results = pool.map_async(make_data, range(5)).get(9999999)
  File "/usr/lib/python3.3/multiprocessing/pool.py", line 562, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[array([[ 0.74628629,  0.36130663, -0.65984794, ..., -0.70921838,
         0.34389663, -1.7135126 ],
       [ 0.60266867, -0.40652402, -1.31590562, ...,  1.44896246,
        -0.3922366 , -0.85012842],
       [ 0.59629641, -0.00623001, -0.12914128, ...,  0.99925511,
        -2.30418136,  1.73414009],
       ..., 
       [ 0.24246639,  0.87519509,  0.24109069, ..., -0.48870107,
        -0.20910332,  0.11749621],
       [ 0.62108937, -0.86217542, -0.47357384, ...,  1.59872243,
         0.76639995, -0.56711461],
       [ 0.90976471,  1.73566475, -0.18191821, ...,  0.19784432,
        -0.29741643, -1.46375835]])]'. Reason: 'error("'i' format requires -2147483648 <= number <= 2147483647",)'

----------
files: multi.py
messages: 185344
nosy: mrjbq7
priority: normal
severity: normal
status: open
title: problem using multiprocessing with really big objects?
versions: Python 3.3
Added file: http://bugs.python.org/file29595/multi.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17560>
_______________________________________