[Python-Dev] Possible GIL/threading issue involving subprocess and PyMem_MALLOC...

[Gregory P. Smith]

On Thu, Dec 20, 2012 at 10:43 AM, Trent Nelson <trent at snakebite.org> wrote:

>     This seems odd to me so I wanted to see what others think.  The unit
>     test Lib/unittest/test/test_runner.py:Test_TextRunner.test_warnings
>     will eventually hit subprocess.Popen._communicate.
>
>     The `mswindows` implementation of this method relies on threads to
>     buffer stdin/stdout.  That'll eventually result in PyOs_StdioReadline
>     being called without the GIL being held.  PyOs_StdioReadline calls
>     PyMem_MALLOC, PyMem_FREE and possibly PyMem_REALLOC.
>

Those threads are implemented in Python so how would the GIL ever not be
held?

-gps


>
>     On a debug build, these macros are redirected to their _PyMem_Debug*
>     counterparts.  The call hierarchy for _PyMem_DebugMalloc looks like
>     this:
>
>         void *
>         _PyMem_DebugMalloc(size_t nbytes)
>         {
>             return _PyObject_DebugMallocApi(_PYMALLOC_MEM_ID, nbytes);
>         }
>
>         /* generic debug memory api, with an "id" to
>            identify the API in use */
>         void *
>         _PyObject_DebugMallocApi(char id, size_t nbytes)
>         {
>             uchar *p;           /* base address of malloc'ed block */
>             uchar *tail;        /* p + 2*SST + nbytes ==
>                                    pointer to tail pad bytes */
>             size_t total;       /* nbytes + 4*SST */
>
>             bumpserialno();
> ------------^^^^^^^^^^^^^^^
>
>             total = nbytes + 4*SST;
>             if (total < nbytes)
>                 /* overflow:  can't represent total as a size_t */
>                 return NULL;
>
>             p = (uchar *)PyObject_Malloc(total);
> -------------------------^^^^^^^^^^^^^^^^^^^^^^^
>             if (p == NULL)
>                 return NULL;
>
>             <snip>
>
>     Both bumpserialno() and PyObject_Malloc affect global state.  The
> latter
>     also has a bunch of LOCK() and UNLOCK() statements, but these end up
> being
>     no-ops:
>
>         /*
>          * Python's threads are serialized,
>          * so object malloc locking is disabled.
>          */
>         #define SIMPLELOCK_DECL(lock) /* simple lock declaration */
>         #define SIMPLELOCK_INIT(lock) /* allocate (if needed) and ... */
>         #define SIMPLELOCK_FINI(lock) /* free/destroy an existing */
>         #define SIMPLELOCK_LOCK(lock) /* acquire released lock */
>         #define SIMPLELOCK_UNLOCK(lock) /* release acquired lock */
>         ...
>         /*
>          * This malloc lock
>          */
>         SIMPLELOCK_DECL(_malloc_lock)
>         #define LOCK()          SIMPLELOCK_LOCK(_malloc_lock)
>         #define UNLOCK()        SIMPLELOCK_UNLOCK(_malloc_lock)
>         #define LOCK_INIT()     SIMPLELOCK_INIT(_malloc_lock)
>         #define LOCK_FINI()     SIMPLELOCK_FINI(_malloc_lock)
>
>     The PyObject_Malloc() one concerns me the most, as it affects huge
>     amounts of global state.  Also, I just noticed PyOs_StdioReadline()
>     can call PyErr_SetString, which will result in a bunch of other
>     calls that should only be made whilst the GIL is held.
>
>     So, like I said, this seems like a bit of a head scratcher.  Legit
>     issue or am I missing something?
>
>         Trent.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20121220/274d0869/attachment.html>