Problem with copy.deepcopy and multiprocessing.Queue

[James DeVincentis]

The problem isn’t with multiprocess.Queue. The problem is that socketserver.ForkingMixIn calls os._exit() after the request is handled by the forked process. Which I did not know until I dug into the internals of socket server.ForkingMixIn.

This causes the Finalize() hooks inside of multiprocess.Queue to not run. The hooks essentially call multiprocess.Queue.close() and multiprocess.Queue.join_thread(). This guarantees the local thread buffer has been flushed before allowing the process to fully exit. Since os._exit() doesn’t honor any of those, the forked process may die without flushing that buffer.  

The documentation is correct for multiprocess.Queue() when it says that multiprocess.Queue.close() get’s run automatically when the thread is garbage collected. It just wasn’t being garbage collected due to the os._exit() call caused by socketserver.ForkingMixIn


> On Oct 18, 2015, at 10:55 AM, Ian Kelly <ian.g.kelly at gmail.com> wrote:
> 
> On Sun, Oct 18, 2015 at 2:46 AM, James DeVincentis <admin at hexhost.net <mailto:admin at hexhost.net>> wrote:
> >
> > I see, looks like I’ll have to use Queue.close()
> >
> > Didn’t think it would be necessary since I was assuming it would be garbage collected. Sigh. Bug, fixed.
> 
> I'm not really following what the issue is here -- it sounds like it runs pretty deep, and I'm not familiar with the module's internals -- but would you please file a bug against the documentation? The docs suggest that multiprocessing.Queue.close probably does not need to be called, and they should be changed if that's not truly the case.
> -- 
> https://mail.python.org/mailman/listinfo/python-list



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20151018/0d5231a5/attachment.html>