using threads with python and winnt

[Tim Peters]

[posted & mailed]

[Tim]
> There's an exceedingly rare timing hole in the released 1.5.2
> Python (fixed in the CVS version) ... showing up in rare reports
> under extremely high rates of thread death and only on top-end
> >multiprocessor systems.

[Aahz Maruch]
> We're still using 1.5.1 (no comments, please! we're supposed
> to finally upgrade this week),

The problem also exists in 1.5.2.

> and I was wondering if you could explain what the symptom of this
> bug is, because we've seen a couple of recent cases where a
> heavily threaded app that creates hundreds of single-shot
> threads dies with a thread error.

The rare msg:

    Fatal Python error: PyThreadState_Delete: invalid tstate

or, even more rarely, a memory fault.  See

http://www.python.org/python-bugs/resolved?id=12;user=guest

for a bit more.

> (We're *also* fixing it to use a thread pool,

That would be an effective workaround; the error is triggered by a race
among threads that are competing to die.  It's like the door to the funeral
home is only wide enough for one, but sometimes two can sneak thru, and then
the casket gets overloaded and everyone at the funeral freaks out when it
bursts open <wink>.

> but I'd still like to know what our chances are of hitting
> this with the stock 1.5.2)

Inevitable, if you repeatedly kill off enough threads over a long enough
time, although the mean-time to failure in non-extreme programs (a dozen
threads spawned every minute or so) would likely work out to no less than
years.  It's much easier to trigger on a true multiprocessor (on a
uniprocessor, a dying thread has to lose its (OS) timeslice in a window of
just a few machine instructions, at the same time another dying thread gets
its timeslice in another window of a few, and they both need to have reached
exactly the same spot in a traversal of a long linked list).

threads-are-unforgiving-ly y'rs  - tim