[ python-Bugs-1041645 ] Thread management corrupts heap
SourceForge.net
noreply at sourceforge.net
Thu Oct 7 02:43:18 CEST 2004
Bugs item #1041645, was opened at 2004-10-07 04:15
Message generated for change (Comment added) made by ncoghlan
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470
Category: None
Group: None
Status: Open
Resolution: None
Priority: 7
Submitted By: benson margulies (benson_basis)
Assigned to: Nobody/Anonymous (nobody)
Summary: Thread management corrupts heap
Initial Comment:
The PyGILState_Ensure mechanism appears to have a
built-in heap-corrupting race condition.
If a new thread calls PyGILState_Ensure, then the code
allocates a new 'tstate' for it on the heap. This
allocation is not protected by any lock. So, multiple
racing threads can hit the heap at the same time, and
corrupt it.
I have observed this with both 2.3 and with 2.4a3.
I will attach a sample application. The application is
Win32, but should be easy enough to adapt to Unix if
someone cares to.
Since the stated purpose of this mechanism, in PEP311,
is to allow any-old-thread to call into Python, I believe
that the usage model here is legitimate.
To watch this explode, run the attached with arguments
like, oh, 1 100 40 against the debug python build.
----------------------------------------------------------------------
Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-07 10:43
Message:
Logged In: YES
user_id=1038590
*rereads bug description and comments*
I think I'm on the right page now. The offending call would
appear to be PyMem_NEW inside PyThreadState_New.
Under a release build, this resolves into a direct call to
malloc(), but under debug it resolves to _PyObjectDebugMalloc.
Which is what you said in the bug report - I just read it
incorrectly. Sorry for the confusion.
I just remembered that this came up a few weeks ago on
Python dev, with a proposed solution (changing this
particular call to use malloc() directly):
http://mail.python.org/pipermail/python-dev/2004-September/048683.html
----------------------------------------------------------------------
Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-07 10:15
Message:
Logged In: YES
user_id=1038590
Disregard the previous comment regarding PyEval_InitThreads
and Py_Initialize - I was reading an older version of the
documentation which suggested the order of the calls mattered.
This does leak the constructed strings (no call to
Py_XDECREF(trash)), but that shouldn't cause anything too
catastrophic.
However, the call to PyGILState_Release without a preceding
call to PyGILState_Ensure looks dubious (it ends up deleting
the main thread's thread state, which seems unhealthy).
Do the same symptoms occur when using the
Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros
around the call to run_group()? (These macros are
recommended when you *know* you already have the GIL. The
PyGILState functions are for when you don't know if you have
the GIL or not)
----------------------------------------------------------------------
Comment By: benson margulies (benson_basis)
Date: 2004-10-07 09:56
Message:
Logged In: YES
user_id=876734
On the one hand, you are correct that changing the order
improves the behavior. On the other hand, the doc is as I
recalled it, asserting that you can call it first. Perhaps this
should be registered as a doc problem?
This is a no-op when called for a second time. It is safe to
call this function before calling Py_Initialize() .
----------------------------------------------------------------------
Comment By: benson margulies (benson_basis)
Date: 2004-10-07 09:49
Message:
Logged In: YES
user_id=876734
The documentation very specifically states that the order
doesn't matter. Further, I've read the code, and the problem
is unprotected data structures in the python debug heap,
which won't be any more protected in the other order.
I'll switch them and report back, just in case.
----------------------------------------------------------------------
Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-07 09:32
Message:
Logged In: YES
user_id=1038590
I haven't checked if that would fix the problem, since I'm
currently on Linux.
----------------------------------------------------------------------
Comment By: Nick Coghlan (ncoghlan)
Date: 2004-10-07 09:31
Message:
Logged In: YES
user_id=1038590
The attached program calls PyEval_InitThreads and
Py_Initialize in the wrong order.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470
More information about the Python-bugs-list
mailing list