Thu Mar 25 18:38:54 CET 2010

M.-A. Lemburg wrote:
> Backporting PyCapsule is fine, but the changes you made to all
> those PyCObject uses does not look backwards compatible.
>
> The C APIs exposed by the modules (e.g. the datetime module)
> are used in lots of 3rd party extension modules and changing
> them from PyCObject to PyCapsule is a major change in the
> module API.

You're right, my changes aren't backwards compatible.  I thought it was 
reasonable for four reasons:

1. The CObject API isn't safe.  It's easy to crash Python 2.6 in just a 
few lines by mixing and matching CObjects.  Switching Python to capsules 
prevents a class of exploits.  I've included a script at the bottom of 
this message that demonstrates three such crashes.  The script runs in 
Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

2. As I just mentioned, Python 3.1 already uses capsules everywhere 
instead of CObjects.  Since part of the purpose of Python 2.7 is to 
prepare developers for the to upgrade to 3.1, getting them to switch to 
capsules now is just one more way they are prepared.

3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we 
ever made a 2.8 I want to remove it completely.

4. When Python publishes an API using a CObject, it describes the thing 
the CObject points to in a header file.  In nearly all cases that header 
file also provides a macro or inline function that does the importing 
work for you.  I changed those to use capsules too.  So if the 
third-party code uses the macro or inline function, all you need do is 
recompile it against 2.7 and it works fine.  Sadly I know of one 
exception: pyexpat.expat_CAPI.  The header file just describes the 
struct pointed to by the CObject, but callers

I can suggest four ways to ameliorate the problem.

First, we could do as Antoine Pitrou suggests on the bug (issue 7992): 
wherever the CObject used to be published as a module attribute to 
expose an API, we could provide both a CObject and a capsule; internally 
Python would only use the capsules.  This would allow third-party 
libraries to run against 2.7 unchanged.  The major problem with this is 
that third-party libraries would still be vulnerable to the 
mix-and-match CObject crash.  A secondary, minor concern: obviously we'd 
store the CObject attribute with the existing name, and the capsule 
attribute would have to get some new name.  But in Python 3.1, these 
attributes already expose a capsule.  Therefore, people who convert to 
using the capsules now would have to convert again when moving to 3.1.

Second, we could make CObject internally support unpacking capsules.  If 
you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and 
return the pointer within.  (We could probably also map the capsule 
"context" to the CObject "desc", if any of the Python use cases needed 
it.)  I wouldn't change anything else about CObjects; creating and using 
them would continue to work as normal.  This would also allow 
third-party libraries to run against Python 2.7 unchanged.  The only 
problem is that it's unsafe, as indeed allowing any use of 
PyCObject_AsVoidPtr() is unsafe.

Third, I've been pondering writing a set of preprocessor macros, shipped 
in their own header file distributed independently of Python and 
released to the public domain, that would make it easy to use either 
CObjects or capsules depending on what version of Python you were 
compiling against.  Obviously, using these macros would require a source 
code change in the third-party library.  But these macros would make it 
a five-minute change.  This could compliment the first or second approaches.

Fourth, we could back out of the changes to published APIs and convert 
them back to CObjects.  -1.

Your thoughts?

/larry/

-----

import sys
def log(message):
    print(message)
    sys.stdout.flush()

def crash1():
    log("Running crash1...")
    try:
        import datetime
        import cStringIO
        cStringIO.cStringIO_CAPI = datetime.datetime_CAPI

        import cPickle
        s = cPickle.dumps([1, 2, 3])
    except ImportError:
        # This test isn't translatable to Python 3.
        pass
    log("Survived crash1!")

def crash2():
    log("Running crash2...")
    try:
        import unicodedata
        import _socket
        _socket.CAPI = unicodedata.ucnhash_CAPI
        import ssl
    except AttributeError:
        # Congratulations, you didn't crash.
        pass
    log("Survived crash2!")

def crash3():
    log("Running crash3...")
    try:
        import unicodedata
        import _multibytecodec
        _multibytecodec.__create_codec(unicodedata.ucnhash_CAPI)

    except ValueError:
        # Congratulations, you didn't crash.
        pass
    log("Survived crash3!")

import sys

if len(sys.argv) > 1:
    if sys.argv[1] == '1':
        crash1()
        sys.exit(0)
    elif sys.argv[1] == '2':
        crash2()
        sys.exit(0)
    elif sys.argv[1] == '3':
        crash3()
        sys.exit(0)

crash1()
crash2()
crash3()