[pyOpenSSL] Python core dumps when using pyOpenSSL 0.8.0 and threads

[Jean-Paul Calderone]

On Mon, 16 Mar 2009 21:43:49 +0100, "M.-A. Lemburg" <mal at egenix.com> wrote:
>It appears as if the new thread lock support in pyOpenSSL 0.8.0 is
>causing serious problems with Python and threads. We have observed
>regular core dumps using pyOpenSSL 0.8 which did not happen with
>0.7.

Ouch.

>There's a patch available on SF which appears to fix the problem:
>
>http://sourceforge.net/tracker/index.php?func=detail&aid=2543118&group_id=31249&atid=401760
>
>Any chance of getting that into a new release ?

A quick skim of the patch leads me to believe it largely reverts one of
the bug fixes I made in 0.8.0 which was causing thread-related crashes
in 0.7.0.  https://bugs.launchpad.net/pyopenssl/+bug/262381 includes
some discussion of that issue.  In the light of my final comment on
that ticket, it may be that the new failure is much worse than the old
failure.

What would make things much easier for me is if someone could provide a
test case which demonstrates the current bug.  The patch which "fixes"
the current problem looks simple enough (it's mostly noise - it could
be 10 lines long instead of a hundred), but since I think it re-introduces
another thread bug, I probably would prefer /not/ to apply it exactly,
but to work out a solution which preserves the old fix and eliminates
the new bug.

If a unit test is too hard, then something along the lines of
leakcheck/thread-crash.py would also be useful.

If that's too hard, an explanation of why the current code is wrong
would help.

>As it stands, version 0.8.0 is not really usable in multi-threaded
>applications.

It seems that 0.7.0 wasn't really either.  And I suspect that without
various vendor-supplied patches that didn't make it upstream until
0.8.0, neither were any of the releases before that one either. :/

Having to revisit this part of pyOpenSSL again, I'm tempted to go for
a solution I previously ruled out for performance reasons - explicitly
tag each OpenSSL object wrapper (for those OpenSSL structures which
can only be used in one thread) with a thread id and before each
operation, check to make sure the current thread matches that thread id.
This is similar to the approach taken by SQLite3, I think.  It should
provide the most robust protection against thread issues, but at the
cost of a thread check per method.

I'm hoping to have 0.9.0 out soon.  Before this issue was raised, I
was only blocked on some issues building Windows packages.  I suppose
it would be good to include some resolution to this issue in the
next release though.  If anyone will be at PyCon and wants to help out,
let me know.

Jean-Paul