[Tutor] threads

[Danny Yoo]

On Wed, 23 Feb 2005, Shitiz Bansal wrote:

> I am trying to build a traffic network simulator using python, for my
> degree project.
>
> I need to run at least 5-6000 cars simultaneously. I wanted to run each
> car in a separate thread. However , after about 400 threads i am unable
> to create new threads.

Hello!

Python is using your operating system's native thread implementation; it's
possible that you're running into a platform-specific issue.


>From looking at the error message:

> Traceback (most recent call last):
>   File "<pyshell#24>", line 2, in -toplevel-
>     i.start()
> error: can't start new thread

I see the error message "error: can't start new thread".  I did a Google
search and found that Windows reports this error if we hit it with
thousands of threads at a time:

    http://mail.python.org/pipermail/python-checkins/2003-July/036869.html

You may need to raise the thread limit on your operating system.  That
being said, doing a simulation through threads might not be the easiest
thing to do.



Instead of representing each car as a thread, you may be able to make
things work by keeping all car in a priority queue, and do a kind of
discrete simulation without threads.

For example, here is a program that simulates people who say that they are
hungry:

######
import heapq
import random

"""Demo of heapq for doing discrete simulation."""


class Scheduler:
    """A scheduler keeps track what the current time is, and what
    event should occur next."""
    def __init__(self):
        self.currentTime = 0
        ## self.queue will be a list of (scheduledTime, callable) tuples.
        self.queue = []


    def after(self, delay, event):
        """Adds a new event to the queue."""
        timeAndEvent = (self.currentTime + delay, event)
        heapq.heappush(self.queue, timeAndEvent)


    def hasNext(self):
        """Returns true if there are still events to be processed."""
        return len(self.queue) > 0


    def next(self):
        """Returns the time and next event to be executed.
        Precondition: we have more events."""
        assert self.queue
        if self.queue:
            timeAndEvent = heapq.heappop(self.queue)
            self.currentTime = timeAndEvent[0]
            return timeAndEvent



class Person:
    """A person knows their name, and what scheduler they belong to."""
    def __init__(self, name, sched):
        self.name, self.sched = name, sched


    def neutralState(self):
        """In a neutral state, a person can either eat or get hungry."""
        print self.name, "is making a choice..."
        nextState = random.choice([self.sleepingState,
                                   self.hungryState])
        self.sched.after(4, nextState)


    def sleepingState(self):
        """A person will wake up in 12 minutes."""
        print self.name, "snores..."
        self.sched.after(12, self.wakeState)


    def wakeState(self):
        print self.name, "wakes up!"
        self.sched.after(1, self.neutralState)


    def hungryState(self):
        print self.name, "is hungry.  Food..."
        self.sched.after(3, self.eatingState)


    def eatingState(self):
        print self.name, "is eating.  Yum."
        self.sched.after(10, self.neutralState)


if __name__ == '__main__':
    scheduler = Scheduler()
    sam = Person("sam", scheduler)
    max = Person("max", scheduler)
    ## Let's start up the initial conditions: sam will be eating, and
    ## max will be sleeping.
    scheduler.after(0, sam.eatingState)
    scheduler.after(0, max.sleepingState)

    for i in range(10):
        time, event = scheduler.next()
        print "time:", time
        event()
######



This is a somewhat hacky program, but it's amusing to watch:

###
time: 0
sam is eating.  Yum.
time: 0
max snores...
time: 10
sam is making a choice...
time: 12
max wakes up!
time: 13
max is making a choice...
time: 14
sam is hungry.  Food...
time: 17
sam is eating.  Yum.
time: 17
max snores...
time: 27
sam is making a choice...
time: 29
max wakes up!
###

It sounds like you want to hit the road with your car simulation; a
discrete simulation approach might be the most straightforward.  If you
make each event discrete enough, it should work ok.



There are systems that support enormous amounts of concurrency; If you
really want to simulate each car with a thread, you may want to try
Stackless Python; it's a Python variant that supports the concept of
lightweight "tasklets".

    http://www.stackless.com/

Also, you may want to also look at Erlang:

    http://www.erlang.org/

Erlang is a programming language that's designed with concurrency in mind,
and I've heard that its concurrency support is second to none.


I hope this helps!