[issue28506] Multiprocessing Pool starmap - struct.error: 'i' format requires -2e10<=n<=2e10

[Justin Ting]

Justin Ting added the comment:

Ah, should have picked that up, coding at 3:30am doesn't do wonders for
keeping a clear head.

Thanks Tim, I'll keep that in mind!

*Justin Ting*
*E* justingling at gmail.com |  *M* +61 424 751 665 | *L*
*https://au.linkedin.com/in/justinyting
<https://au.linkedin.com/in/justinyting>* | *G *https://github.com/jyting

On Sun, Oct 23, 2016 at 3:48 AM, Tim Peters <report at bugs.python.org> wrote:

>
> Tim Peters added the comment:
>
> This has nothing to do with the _values_ you're passing - it has to do
> with the length of the pickle string:
>
>     def _send_bytes(self, buf):
>         n = len(buf)
>         # For wire compatibility with 3.2 and lower
>         header = struct.pack("!i", n)  IT'S BLOWING UP HERE
>         if n > 16384:
>             ...
>             self._send(header)
>             self._send(buf)
>
> where the traceback shows it's called here:
>
>     self._send_bytes(ForkingPickler.dumps(obj))
>
> Of course the less data you're passing, the smaller the pickle, and that's
> why it doesn't blow up if you pass subsets of the data.
>
> I'd suggest rethinking how you're sharing data, as pushing two-gigabyte
> pickle strings around is bound to be the least efficient way possible even
> if it didn't blow up ;-)
>
> ----------
> nosy: +tim.peters
>
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue28506>
> _______________________________________
>

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28506>
_______________________________________