[issue29423] using concurrent.futures.ProcessPoolExecutor in class giving 'TypeError: can't pickle _thread.RLock objects' in 3.6, but not 3.5

[Josh Rosenberg]

Josh Rosenberg added the comment:

Looks like the problem here is that pickling self._consume implicitly pickles self, and after the first submission, self._futures contains Future objects. Those are probably what have the RLock in them (to synchronize between reading and populating threads), and it's correct not to pickle them; they're only meaningful in the parent process, and sending them to the workers would break, badly.

Given you don't seem to be using self for anything, you could either make _consume a staticmethod (if you need state from self, pass that specific state in as arguments) so it's not implicitly pickling and transmitting self, or failing that, define custom pickling functions that explicitly exclude self._futures from the set of attributes to pickle.

There is nothing to fix behaviorally on Python's side that wouldn't lead to confusion/breakage in other code. It would be nice if Python could give more detail on pickle chain that led to the unpicklable object (so you know it's self._consume->self->self._futures->some future object->future._rlock or whatever), but I'm not sure the pickling protocol design can preserve that sort of chain for error reporting...

----------
nosy: +josh.r

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue29423>
_______________________________________