[C++-sig] Boost.Python v2: reference-counting and user-friendly conversions

[David Abrahams]

So handle<> went into the codebase as a replacement for reference<> last
night. I hope it didn't cause too much pain. Contradicting my earlier
suggestion, and because Python  so commonly passes back new references, the
protocol for reference-counting remained the same as with reference<>: the
constructor does not increment the reference count of its argument by
default. Also, at least for the time being, the default behavior is to
throw an exception if it is initialized with a zero pointer. This might be
changed once the "object" class, discussed below, is finished.

The interface for causing the reference count to be incremented has
changed: now one passes the argument through the borrow() function,
signaling that we are getting a borrowed reference:

    handle<> x(borrow(PyTuple_GetItem(t, n)));

The interface for accepting null pointer arguments without throwing an
exception is similar:

    handle<> x(allow_null(PyObject_GetAttrString(o, name)));


Now, on to user-friendly conversions. It seems to me that the best way to
spell to_python(x) would be simply:

    object(x)

That is, object would have a templated explicit constructor which performs
the conversion. But don't be distracted by this, it doesn't change any of
the issues below (we have all the same problems if we spell it object x =
to_python(x)).

So, what's the behavior when the user passes a raw PyObject*? It seems to
me that we'd like the following to work:

    object(PyObject_GetAttrString(some_PyObject_pointer,
"some_attribute_name"))

Which implies, of course, that object has the same reference-counting
behavior as handle<>. However, the explicit to_python conversion behavior
is currently defined to be the same as that which is used for arguments to
call<> and call_method<>. Let's take a look at what's defined there:

Right now, if you pass a PyObject* as an argument to call<>, we first check
for NULL (in which case we manufacture a reference to None) then the
reference count is incremented before the arg_to_python converter takes
possession.

So that contradicts the desired semantics for object construction. Now, I'm
less-than-confident that the NULL => None behavior for arguments to call<>
is really preferable to throwing an immediate exception when NULL is
passed, however, it seems obvious to me that we don't want call<> stealing
references from arbitrary PyObject*s. I think this just goes to prove once
again that manual reference-counting is nasty.

There's also the issue of converting other raw Python object pointers, of
which there are two varieties:

A. Pointers to C++ classes derived from PyObject
B. Pointers to POD structs which are layout-compatible with PyObject, e.g.
PyTypeObject*. These are pseudo-derived types.

I can reliably detect A above, but very few compilers can detect B, and
then it's a crapshoot (you can look for a publicly-accessible ob_refcnt
member of the right type, for example). We could work around that with a
traits class that the user explicitly specializes for each "pseudo-derived"
type he wants to use.

Questions:

1. How should we handle raw Python object arguments to
call<>/call_method<>?
    a. is NULL->None conversion a good idea for these particular pointers?
       I still think it makes sense for most pointers, but maybe not these.
       see
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/boost/boost/libs/
python/doc/v2/callbacks.html
       for a review of semantics

    b. is there any reason to think that stealing semantics should be
       introduced, so that:

            call<void>(my_function, PyObject_GetAttrString(x, y))

       would work correctly? Remember that we will be providing wrappers
for
       the Python API functions. Also remember that my_function is
currently
       a PyObject* which is /not/ stolen.

    c. Is it important to be able to accept variety A and B of
       raw Python object pointer as arguments here? Remember that we will
       have object and handle<> whose refcount semantics are unambiguous

    d. Is it reasonable to require a traits specialization or other action
       from the user to explicitly denote variety B? We can neither
generate
       a compile-time error nor special handling if we can't detect them.

2. How should we handle raw Python object arguments to the object()
constructor?
    a. is NULL->None conversion a good idea for these particular pointers?

    b. is there any reason to think that stealing semantics should not be
       used?

    c. Is it important to be able to accept variety A and B of
       raw Python object pointer as arguments here? Remember that we will
       have object and handle<> whose refcount semantics are unambiguous

Thanks for your feedback,
Dave

+---------------------------------------------------------------+
                  David Abrahams
      C++ Booster (http://www.boost.org)               O__  ==
      Pythonista (http://www.python.org)              c/ /'_ ==
  resume: http://users.rcn.com/abrahams/resume.html  (*) \(*) ==
          email: david.abrahams at rcn.com
+---------------------------------------------------------------+