[issue17560] problem using multiprocessing with really big objects?

[Richard Oudkerk]

Richard Oudkerk added the comment:

> I *think* we need to keep compatibility with the wire format, but perhaps 
> we could use a special length value (-1?) to introduce a longer (64-bit) 
> length value.

Yes we could, although that would not help on Windows pipe connections (where byte oriented messages are used instead).  Also, does pickle currently handle byte strings larger than 4GB?

But I can't help feeling that multigigabyte arrays should be transferred using shared mmaps rather than serialization.  numpy.frombuffer() could be used to recreate the array from the mmap.

multiprocessing currently only allows sharing of such shared arrays using inheritance.  Perhaps we need a picklable mmap type which can be sent over pipes and queues.  (On Unix this would probably require fd passing.)

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue17560>
_______________________________________