Python vs. Ruby: threads - advice appreciated

[Tim Peters]

[posted & mailed]

[Jean-Paul Smets]
> I am currently comparing Python and Ruby for a personal project
> which requires many many threads.
>
> I came across the "philosophers" example in Ruby and was surprised by its
> compacity (see below).
> I am wondering if it would be easy to write such an example in
> Python with as few lines of code ?

Fewer (Python doesn't require the noise "end" lines), but I won't show
that -- approaching a large-thread project with masses of global vrbls and
repeated inline fiddly arithmetic (to map integer ids to conceptual objects
to conceptual properties of those objects via global arrays) is a dead end
for maintainability.  Instead I'll attach a Python example that uses its
higher-level objected-oriented thread interface:  treat a Philospher as an
object, and then, e.g., it's natural and easy to give each Philospher named
left_fork and right_fork objects, greatly increasing the readability of the
code after initial setup.

> Also, I tried to push Ruby to its limits (N=10000) and everything
> went OK (50 Mo RAM was occupied though but this is only 5Ko per
> thread which I find quite good).

This almost certainly has much more to do with the operating system you're
running on than with the language you use.  A Python thread requires trivial
overhead beyond what the underlying platform thread requires; don't know
about Ruby, but suspect it's much the same there (if a language supplies a
native thread, it can't require *less* burden than a platform thread
demands!).  Note that if your 5Kb/thread is accurate, and it includes space
for the thread stack, your platform's threads are going to blow up if a
thread does deep function calls (indeed, on most platforms it's a thread's
stack that accounts for the bulk of a thread's memory burden).

> I wonder if it is possible to go that far with Python, either with
> native threads

You didn't name your OS, so nobody can try a comparable experiment.  The
Python below invites 1,000 Philosophers to dinner, and ran fine without
thrashing on an old 32Mb Win95 box.  I know from experience I can't push
this particular box much farther than that.

> or with microthreads in stackless Python ?

A Stackless microthread has much less overhead than a native thread (largely
because it doesn't have to pre-reserve space for "a stack"), but note that
Stackless is not part of the std Python distribution.

> Finaly, I found that Ruby has a very nice and completely
> transparent interface to Python libraries and extensions. So,
> I am wondering now what would be the inconvenience for me of
> switching to Ruby for my project.

Beats me.  Even if I understood the Ruby code, I would find the Python below
much more readable, and for a large project that's of overwhelming
importance to me.  I don't know whether Ruby can be made comparably easy to
live with, but suspect it probably can be.

[the Ruby example]
> #
> # The Dining Philosophers - thread example
> #
> require "thread"
>
> srand
> #srand
> N=10				# number of philosophers
> $forks = []
> for i in 0..N-1
>   $forks[i] = Mutex.new
> end
> $state = "-o"*N
>
> def wait
>   sleep rand(20)/10.0
> end

I personally hate this Perl-like "function calls don't require parens"
gimmick.  In Perl it can kill you.  In Ruby, I don't know whether to parse
that line as

    sleep(rand(20)/10.0)

or

    sleep(rand(20))/10.0

although from *context* only the first makes any sense.

> def think(n)
>   wait
> end
>
> def eat(n)
>   wait
> end
>
> def philosopher(n)
>   while TRUE
>     think n
>     $forks[n].lock
>     if not $forks[(n+1)%N].try_lock
>       $forks[n].unlock		# avoid deadlock
>       next
>     end
>     $state[n*2] = ?|;
>     $state[(n+1)%N*2] = ?|;
>     $state[n*2+1] = ?*;
>     print $state, "\n"

Surely this is buggy:  unless Ruby has some implicit global function lock
that serializes calls to philosopher, multiple threads can be mucking with
the global $state string simultaneously, and so the output can be
inconsistent.

>     eat(n)
>     $state[n*2] = ?-;
>     $state[(n+1)%N*2] = ?-;
>     $state[n*2+1] = ?o;
>     print $state, "\n"

Ditto.

>     $forks[n].unlock
>     $forks[(n+1)%N].unlock
>   end
> end
>
> for i in 0..N-1
>   Thread.start{philosopher(i)}
>   sleep 0.1
> end

What purpose does that sleep serve?

> sleep

Ditto.

overkill-attached-ly y'rs  - tim

import threading, random, time

# Number of philosophers.
N = 1000

# Number of times a philosopher needs to eat.
# Letting the program run forever is cute, but there isn't
# that much food in the universe <wink>.
EATME = 10

def wait(atmost=2):
    """Wait a random time, by default at most 2 seconds."""
    time.sleep(random.random() * atmost)

# Exception raised to let a philosopher know that she can't
# get both forks.
class ForksBusy(Exception):
    pass

class Philosopher(threading.Thread):
    def __init__(self, id):
        threading.Thread.__init__(self)
        self.id = id
        self.left_fork = forks[id]
        self.right_fork = forks[(id + 1) % N]
        self.feedings = 0

    def run(self):
        # Compare this main loop:  using OO organization, the
        # logic is much less cluttered, and so correspondingly
        # easier to understand and extend.
        while self.feedings < EATME:
            try:
                self.think()
                self.acquire_forks()  # may raise ForksBusy
                self.eat()
                self.release_forks()
            except ForksBusy:
                pass
        print "philosopher %d stuffed" % self.id

    def think(self):
        print "philosopher %d thinking" % self.id
        wait()

    def eat(self):
        self.feedings = self.feedings + 1
        print "philosopher %d on feeding #%d" % (self.id,
                                                 self.feedings)
        wait()

    def acquire_forks(self):
        self.left_fork.acquire()
        if not self.right_fork.acquire(0):
            self.left_fork.release()
            raise ForksBusy

    def release_forks(self):
        self.left_fork.release()
        self.right_fork.release()

# Create the forks.
forks = []
for i in range(N):
    forks.append(threading.Lock())

# Create the philosophers, and start their contemplative lives.
philosophers = []
for i in range(N):
    p = Philosopher(i)
    philosophers.append(p)
    p.start()

# Wait for everyone to finish.
for p in philosophers:
    p.join()

print "Dinner is complete."