Multiprocessing / threading confusion

[Piet van Oostrum]

Paul Pittlerson <menkomigen6 at gmail.com> writes:

> On Friday, September 6, 2013 1:46:40 AM UTC+3, Chris Angelico wrote:
>
>> The first thing I notice is that your Debugger will quit as soon as
>> its one-secondly poll results in no data. This may or may not be a
>> problem for your code, but I'd classify it as code smell at best. Is
>> your goal here to make sure Debugger doesn't stop your process from
>> exiting? If so, a simpler and safer solution is to make it a daemon
>> thread.
>
> I didn't think it would be a problem, because unless the system is very
> slow, the functions will finish in a fraction of a second, on my machine
> it does not matter whether I have it set as 0.1 second or several seconds,
> the output is still the same. It's not eloquent, but the point was just to
> exit the test when no more prints are to be made.
>
> But how can I fix the actual bug I was asking about though? I want to 
> print ticked and exited for all the processes, just to acknowledge to
> myself that the code is working.. so I can proceed to experiment with
> more complexity! :D

On my system I get the output:

started worker 75501
75501 ticked
75501 has exited
started worker 75505
75505 ticked
75505 has exited
started worker 75504
started worker 75502
started worker 75503
75502 ticked
75502 has exited
75504 ticked
75504 has exited
75503 ticked
75503 has exited

So all the provesses have their 'started' 'ticked' and 'exited' message.
But as others have indicated, because your code is timing dependent,
that is just coincidental. Because multiprocessing/multithreading is
inherently non-deterministic, the order of the messages will be
unpredictable. But you should not make the exiting of the Debugger non
deterministic, as you have it now. One way to do this is to count the
number of processes that have exited, and wait until all are done. In
this case you could count the number of 'exited' messages that have
arrived.

    def run(self):
        nbr_process = 5
        while True:
            time.sleep(1)
            
            msg = self.q.get()
            print msg
            if 'exited' in msg:
                nbr_process -= 1
                if nbr_process == 0:
                    break
 
Of course the 5 should be given as a parameter.

This still leaves you with the uncertainty of the __del__ being called.
Why not just put the message at the end of the Worker run code?

    def run(self):
        self.que.put('%s ticked' % self._pid)
        # do some work
        time.sleep(1)
        self.que.put('%s has exited' % self._pid)

However, in my experiments (bot Python 2.7.5 and 3.3.2 on Mac OS X
10.6.8) it seems that there is a problem with a Thread inside a Process:
When the main thread of the Process is finished, the other Thread is
also terminated, as if it is a daemon thread. Although self.daemon ==
False!!

You can check this with the following Worker code:

    def run(self):
        for n in range(5):
            self.que.put('%s tick %d' % (self._pid, n))
            # do some work
            time.sleep(1)
        self.que.put('%s has exited' % self._pid)

It appears that not all ticks are deliverd. In my system, only one tick
per thread, and then it disappears. I have no idea if this is a bug. I
certainly couldn't find it documented.

The solution to this is to put a join statement in gogo:

def gogo(qu):
    w = Worker(qu)
    w.start()
    w.join()

-- 
Piet van Oostrum <piet at vanoostrum.org>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]