[Python-Dev] Strange segfault in Python threads and linux kernel 2.6

[Michael Hudson]

Donovan Baarda <abo at minkirri.apana.org.au> writes:

> On Wed, 2005-01-19 at 13:37 +0000, Michael Hudson wrote:
>> Donovan Baarda <abo at minkirri.apana.org.au> writes:
> [...]
>> You've left out a very important piece of information: which version
>> of Python you are using.  I'm guessing 2.3.4.  Can you try 2.4?
>
> Debian Python2.3 (2.3.4-18), Debian kernel-image-2.6.8-1-686 (2.6.8-10),
> and Debian kernel-image-2.4.27-1-686 (2.4.27-6)
>
>> I'd be astonished if this is the same bug.
>> 
>> The main oddness about python threads (before 2.3) is that they run
>> with all signals masked.  You could play with a C wrapper (call
>> setprocmask, then exec fop) to see if this is what is causing the
>> problem.  But please try 2.4.
>
> Python 2.4 does indeed fix the problem. 

That's good to hear.

> Unfortunately we are using Zope 2.7.4, and I'm a bit wary of
> attempting to migrate it all from 2.3 to 2.4. 

That's not so good to hear, albeit unsurprising.

> Is there any way this "Fix" can be back-ported to 2.3?

Probably not.  It was quite invasive and a bit scary.  OTOH, it hasn't
been the cause of any bug reports yet, so it can't be all bad.

> Note that this problem is being triggered when using 
> Popen3() in a thread. Popen3() simply uses os.fork() and os.execvp().
> The segfault is occurring in the excecvp'ed process. I'm sure there must
> be plenty of cases where this could happen. I think most people manage
> to avoid it because the processes they are popen'ing or exec'ing happen
> to not use signals.

Indeed.

> After testing a bit, it seems the fork() in Popen3 is not a contributing
> factor. The problem occurs whenever os.execvp() is executed in a thread.
> It looks like the exec'ed command inherits the masked signals from the
> thread.

Yeah.  I could have told you that, sorry :)

> I'm not sure what the correct behaviour should be. The fact that it
> works in python2.4 feels more like a byproduct of the thread mask change
> than correct behaviour. 

Well, getting rid of the thread mask changes was one of the goals of
the change.

> To me it seems like execvp() should be setting the signal mask back
> to defaults or at least the mask of the main process before doing
> the exec.

Possibly.  I think the 2.4 change -- not fiddling the process mask at
all -- is the Right Thing, but that doesn't help 2.3 users.  This has
all been discussed before at some length, on python-dev and in various
bug reports on SF.

In your situation, I think the simplest thing you can do is dig out an
old patch of mine that exposes sigprocmask + co to Python and either
make a custom Python incorporating the patch and use that, or put the
code from the patch into an extension module.  Then before execing
fop, use the new code to set the signal mask to something sane.  Not
pretty, particularly, but it should work.

>> > BTW, built in file objects really could use better non-blocking
>> > support... I've got a half-drafted PEP for it... anyone interested in
>> > it?
>> 
>> Err, this probably should be in a different mail :)
>
> The verboseness of the attached test code because of this issue prompted
> that comment... so vaguely related :-)

Oh right :) Didn't actually read the test code, not having fop to
hand...

Cheers,
mwh

-- 
  The ability to quote is a serviceable substitute for wit.
                                                -- W. Somerset Maugham