Critical sections and mutexes

[David Brady]

> -----Original Message-----
> From: brueckd at tbye.com [mailto:brueckd at tbye.com]
> Sent: Tuesday, October 23, 2001 7:33 PM
> Subject: Re: Critical sections and mutexes
> 
> No, "normal" operations on Python objects 
> are atomic as far as threads are concerned.
[snip]
> Consider a simple producer/consumer
> situation:

Yes, but.

Consider this minor rewriting of your code.  Three
changes have been made.  First, I made it a stack
instead of a list, so insertions and removals are
happening at the same point.  Second, I do a decidedly
non-atomic set of operations: I peek at the stack,
print a message, then pop the stack.  Finally, I
tightened the timing down to 0.1 seconds max to
increase the likelihood of a collision.

It took about 6 runs, but I finally got one:

#----------------------------------------
# Code
import threading, time, random

foo = []

def producer():
    for i in range(10):
        print 'Producing', i
        foo.insert(0,i)
        time.sleep(random.random() * 0.1)
        print 'Producer done'

def consumer():
    count = 0
    while 1:
        try:
            peeknum = foo[0]
            print 'Consumer: about to consume %d.' %
peeknum
            num = foo.pop(0)
            print 'Consuming', num
            count += 1
            if count >= 10:
                break
        except IndexError:
            pass
        time.sleep(random.random() * 0.1)
        print 'Consumer done'

threading.Thread(target=consumer).start()
threading.Thread(target=producer).start()


"""
Output: Notice last 2 lines
...
Producing 3
Consumer: about to consume 3.
Consuming 3
Producing 4
Producing 5
Consumer: about to consume 4.
Consuming 5
...
"""

Now, this is arguably a malicious rewriting of your
code, and it certainly flies in the face of the notion
that multithreaded code should at least be careful
about how it interacts with its neighbors.

Anyway, this is the kind of "critical section" idea
I'm talking about.  If I do a series of operations
expecting a shared resource to remain unchanged, I
either need to have uninterruptable code (a critical
section) or the resource needs to be mutexed.

Interestingly, in several sets of tests, I discovered
that print statements aren't entirely atomic; tightly
threaded code can interrupt a print statement after
the text, but before the newline has been printed:

def fn1():
    for i in range(10):
        print "Hello Bob!"
        time.sleep(random.random() * 0.1)

def fn2():
    for i in range(10):
        print "This is Fred!"
        time.sleep(random.random() * 0.1)

...can produce output like this:
Hello Bob!
This is Fred!
Hello Bob!This is Fred!

Hello Bob!
This is Fred!

Anyway.  The upshot is, I need to think about Python
threading in terms of mutices [1] instead of critical
sections; groups of resources that need to be accessed
atomically will need a mutex to control them.

For the nonce, I have rewritten my PriorityQueue class
to use a pair of Queue.Queues, and included a mutex
above them so that two threads cannot simultaneously
access the separate queues.  I still need to profile
it, but I'm assuming that Queue.Queue will be more
efficient over the long haul than a list.  My
assumption is that Queues are fairly smart about
managing memory creep, but the queue's mutex code may
be an unnecessary speed hit.  I'll know for sure once
I profile them; it may be that lists are better.  (The
reality is, both are probably sufficient.)

Thank you all for your responses.

-dB
[1] Yeah, yeah.  I know it's not a word... but neither
is "Kleenices", and I say that, too.  :-)

=====
David Brady
daves_spam_dodging_account at yahoo.com
I'm feeling very surreal today... or *AM* I?

__________________________________________________
Do You Yahoo!?
Make a great connection at Yahoo! Personals.
http://personals.yahoo.com