Python or OS forking/threading problem?

[Neil Schemenauer]

Andrew M. Kuchling <akuchlin at mems-exchange.org> wrote:
>It doesn't crash on Solaris; I increased the number of forking
>processes to 10.  I wonder if all those threads might be causing so
>much scheduler overhead that it only looks like a deadlock on Linux,
>but really it's just spending lots of time in the kernel trying to
>pick the next process to run.

The CPU usage is 100% but only a small percentage is system time.
The scheduling should show up as system time shouldn't it?.  I
don't think sleep() or wait() is the problem either, Python seems
to hang in the same place quite consistently.  The only task
running (using 100% cpu) is doing this:

    pthread_cond_wait () from /lib/libpthread.so.0
    PyThread_acquire_lock at thread_pthread.h:318
    PyEval_AcquireThread at ceval.c:150
    t_bootstrap at ./threadmodule.c:223
    pthread_start_thread () from /lib/libpthread.so.0

None of the threads are in a call to sleep(), wait() or fork().
This comment from the pthread_atfork man page seems like it
_might_ be relevent:

    To  understand  the purpose of pthread_atfork, recall that
    fork(2)  duplicates  the  whole  memory  space,  including
    mutexes in their current locking state, but only the call­
    ing thread: other threads are not  running  in  the  child
    process.  Thus, if a mutex is locked by a thread other than
    the thread calling fork, that  mutex  will  remain  locked
    forever in the child process, possibly blocking the execu­
    tion of the child process. 

I don't see how this could cause a problem for Python though.
AFAIK, Python only using one lock.  The thread that has that lock
has to be the one that is calling fork().

If someone has any theories about what is happening here I would
be eager to hear them.


    Neil


-- 
Real programmers don't make mistrakes