Python or OS forking/threading problem?

[Andrew M. Kuchling]

nascheme at enme.ucalgary.ca (Neil Schemenauer) writes:
> I don't know much about threading but here is my small
> contribution.  The attached program locks up on my machine when I
> increase the number of forking processes to more than one.  It
> happens for the CVS version of Python as well as 1.5.2.

It doesn't crash on Solaris; I increased the number of forking
processes to 10.  I wonder if all those threads might be causing so
much scheduler overhead that it only looks like a deadlock on Linux,
but really it's just spending lots of time in the kernel trying to
pick the next process to run.

>Also, there doesn't seem to be major changes to threading.py,
>posix.waitpid or threadmodule.c between 1.5.2 and the current CVS
>source.  Andrew, are you sure it doesn't lock up on Solaris?
> Maybe you need to increase the number of threads.

Pretty sure, I think; I ran it with 50 threads, and while it was
pretty slow (scheduling overhead, I imagine), the program *did*
complete.  I thought it might be some change to signal handling that's
responsible for the fix, so that different treatment of SIGCHLD was
responsible, but there are no relevant changes since 1.5.

Unless ... I noticed something suspicious along the way; look at this
code from floatsleep() in Modules/timemodule.c:

	Py_BEGIN_ALLOW_THREADS
	if (select(0, (fd_set *)0, (fd_set *)0, (fd_set *)0, &t) != 0) {
		Py_BLOCK_THREADS
#ifdef EINTR
		if (errno != EINTR) {
#else
		if (1) {
#endif
			PyErr_SetFromErrno(PyExc_IOError);
			return -1;
		}
	}
	Py_END_ALLOW_THREADS

Py_BLOCK_THREADS is for leaving a {BEGIN,END}_ALLOW_THREADS block (see
ceval.h), but this code doesn't always exit; if errno == EINTR, the
flow would be Py_BEGIN_ALLOW_THREADS; Py_BLOCK_THREADS;
Py_END_ALLOW_THREADS.  I suspect the BLOCK_THREADS should be moved to
inside the if, so it's only executed when the function actually
returns unexpectedly.  But I don't know if this might be the root of
the problem.

Any threading wizards such as Tim Peters or Greg Stein want to offer
some insight, whether into this possible bug or the original problem?

-- 
A.M. Kuchling			http://starship.python.net/crew/amk/
Science itself, therefore, may be regarded as a minimal problem, consisting of
the completest possible presentment of facts with the least possible
expenditure of thought.
    -- Ernst Mach