[Python-Dev] A question concerning named pipes, multiprocessing.Queue (multiprocessing.queues.JoinableQueue)

Matt Gregory skyleach at gmail.com
Mon Aug 1 11:50:38 EDT 2016


I raised this issue and question on StackExchange and #python (FreeNode) 
and have received little or no feedback.  I fear that the only answer 
will lie in profiling the python interpreter itself, which is beyond the 
scope of my capabilities at present.

The original question can be found here: 
http://stackoverflow.com/questions/38637282/multiprocessing-queue-seems-to-go-away-os-pipe-destruction-vs-python

Cliffs: after about one to two hours of processing, the consumer process 
reading the queue terminates on timeout during queue.get(). The thread 
writing objects to the queue receives no exception continues writing to 
the queue.  The thread keeps track of each child process using a simple 
local list of processes, and each time an object is added to the queue, 
the processes objects (consumers) are checked with .is_alive().  When 
the child processes suicide on timeout, is_alive() continues to return 
True, so they are not garbage collected nor are new processes started.

I sincerely apologize if my own understanding of the documentation is at 
fault, but to the best of my knowledge this code should work according 
to the documentation.


More information about the Python-Dev mailing list