[Python-Dev] PEP 384 status

[Nick Coghlan]

On Wed, Sep 1, 2010 at 7:49 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> Yes, and it's a pretty common situation.   The fopen() that I call within a
>> DLL may not be the same fopen() called by another DLL.  When writing a DLL
>> for Windows, the API must be designed with the assumption that anything
>> returned by the C library cannot be passed a different C library.  For
>> example, suppose I a expose a function in my DLL that allocates some memory,
>> populates it with useful information, and returns a pointer.  I must also
>> supply a function to free the memory.  I cannot ask the caller to simply
>> call free(), because their free() may not be using the same heap as my
>> malloc().
>
> But isn't exactly that a major problem for Python ?
>
> An extension written for a different MS CRT version would
> not be able to safely free a buffer allocated by the Python
> DLL.

Correct, but we generally don't allow raw pointers across the API
boundary like that. Instead, users are told to release memory with
PyMem_Free, PyObject_Delete, reference decrementing, etc. That means
the memory release happens in the runtime linked with the Python DLL
and everything is fine.

>
>> Likewise, a FILE * isn't safe to pass around, unless I can guarantee that
>> the application really is one big happy family compiled against the same
>> version of the C library.
>
> According to the MS document I found this also applies to the OS
> environment and handles.
>
> Taking all these things together makes it sound like there's
> a rather small chance of actually getting PEP 384 working
> across Windows compiler upgrades.

Not really - we just need to keep an eye out for the places where
those things can leak through and ensure they're instead abstracted so
the extension module only talks to Python rather than being given
direct access to the underlying C runtime details. FILE* is one, errno
is another, locale we may just have to live with (alternatively, we
could add a callback API to allow extension modules to be notified
when the locale changes, which may be useful for other reasons... see
issue9727). Antoine pointed out file descriptors as another possible
problem, but I'm not sure we can prevent that at the API level (there
are too many ways for Python code to get at file descriptors - it is
likely better to just say that if you get a file descriptor from
Python, only use it with Python APIs, not directly with the C
runtime).

> Then again, I haven't heard of many people actually running into
> those possible issues.

Even today you can use a different C runtime in a Python extension so
long as you avoid the APIs that would cause problems (e.g. I have
legacy SWIG-wrapped extensions built with VC6 that run quite happily
even when linked against Python 2.4).

The idea with the stable ABI is to explicitly exclude those functions
so that you *know* you aren't going to run into those problems.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia