[Python-Dev] Changing pymalloc behaviour for long running processes

[Michael Hudson]

Tim Peters <tim.peters at gmail.com> writes:

> In theory, the calling thread holds the GIL (global interpreter lock)
> whenever an obmalloc function is called.  That's why the lock macros
> inside obmalloc expand to nothing (and not locking inside obmalloc is
> a significant speed win).
>
> But in some versions of reality, that isn't true.  The best available
> explanation is in new_arena()'s long internal comment block:  because
> of historical confusions about what Python's memory API *is*, it's
> possible that extension modules outside the core are incorrectly
> calling the obmalloc free() when they should be calling the system
> free(), and doing so without holding the GIL.  At the time obmalloc
> last got a rework, we did find some extensions that were in fact
> mixing PyObject_{New, NEW} with PyMem_{Del, DEL, Free, FREE}. 
> obmalloc endures extreme pain now to try to ensure that still works,
> despite the lack of proper thread locking.  As the end of that comment
> block says,
>
> * Read the above 50 times before changing anything in this
> * block.
>
> Now all such insane uses have been officially deprecated, so you could
> be bold and just assume obmalloc is always entered by a thread holding
> the GIL now.  I don't know whether it's possible to get away with
> that, though -- if some "important" extension module is still careless
> here, it will break in ways that are all of catastrophic, rare, and
> difficult to reproduce or analyze,  If I could make time for this, I'd
> risk it (but for 2.5, not for 2.4.x), and proactively search for-- and
> repair --external extension modules that may still be insane in this
> respect.

Would it be possible to (in a debug build, presumably) do

assert(I have the GIL);

in PyObject_Free?

Cheers,
mwh

-- 
  If you don't have friends with whom to share links and conversation,
  you have social problems and you should confront them instead of
  joining a cultlike pseudo-community.    -- "Quit Slashdot.org Today!"
   (http://www.cs.washington.edu/homes/klee/misc/slashdot.html#faq)