Processes not exiting

[ma3mju]

On 31 July, 11:34, MRAB <pyt... at mrabarnett.plus.com> wrote:
> ma3mju wrote:
> > Hi all,
>
> > I'm having trouble with multiprocessing I'm using it to speed up some
> > simulations, I find for large queues when the process reaches the
> > poison pill it does not exit whereas for smaller queues it works
> > without any problems. Has anyone else had this trouble? Can anyone
> > tell me a way around it? The code is in two files below.
>
> [snip]
>
> > #get number of cores and set the number on concurrent processes
> > num_hard_workers = 2
> > num_workers = multiprocessing.cpu_count()*1.5
> > easy_workers = []
> > hard_workers = []
> > #add poison pill for each worker and create the worker
> > for i in range(0,num_workers-num_hard_workers):
> >     easy_work_queue.put(None)
> >     easy_workers.append(multiprocessing.Process
> > (target=GP.RandomWalkGeneralizationErrorParallel,args=
> > (easy_work_queue,result_queue,)))
> > for i in range(0,num_hard_workers):
> >     hard_work_queue.put(None)
> >     hard_workers.append(multiprocessing.Process
> > (target=GP.RandomWalkGeneralizationErrorParallel,args=
> > (hard_work_queue,result_queue,)))
>
> You have 2 hard workers and ceil(CPU_count * 1.5) - 2 easy workers.
> What if the number of CPUs was 1? That would give 2 hard and 0 easy!
>
> Also, I recommend that you put only 1 'poison pill' in each queue and
> have the workers put it back when they see it.

I'll give that a go in a sec and see if it helps. The processes quit
out for smaller queues though so it should in theory be alright. I'm
not too fussed about the CPU's it's only there because I change
between a uni PC and home one with a different number of cores in each
but both greater than one.