From sharris at primus.com  Fri Jun  2 00:04:06 2000
From: sharris at primus.com (Steve Harris)
Date: 01 Jun 2000 15:04:06 -0700
Subject: [C++-SIG] (Still) confused about wrapping C++ types with CXX
Message-ID: <svd7m16otl.fsf@hodge.primus.com>

I've spent almost a week trying to figure out how to ask my
question(s) and am still having difficulty finding a concise way to
share the issues I'm facing. Here goes:

I'm trying to prove that I can write non-intrusive Python wrappers
around non-Python-aware C++ types. In order to start in on this
exercise, I created a hypothetical C++ type called "some_pod":

==================================================
#ifndef POD_HH
#define POD_HH

#include <string>
#include <list>
#include <map>


class some_pod
{
public:
    some_pod(const std::string&);
    ~some_pod();

    some_pod(const some_pod&);
    some_pod& operator=(const some_pod&);

    void do_it() const;
    const std::list<int>& get_list() const;
    const std::map<std::string, int>& get_map() const;

private:
    std::string m_str;
    std::list<int> m_list;
    std::map<std::string, int> m_map;
};


#endif  // POD_HH
==================================================

some_pod is not exactly POD in the rigorous sense, but it embodies a
pattern common to the classes I'll need to wrap with Python: it does
some stuff, and exposes collections of other stuff. Sometimes these
collections contain items that are themselves collections. I didn't
show any "setter" methods here, or collection "getters" that expose
non-const references. Eventually I'll need to tackle those as well.

One of the first things I did was to write some support classes to
enable me to write a Py::PythonExtension-derived wrapper around a live
instance of some_pod (or any other type used as a template
argument). That is, my Py::PythonExtension-derived class does not
contain a some_pod instance; it contains a reference to one. The
assumption there is that the some_pod instance would likely be "live"
in the hosting application, and that application will want to call
into some user-provided Python code, offering up the some_pod instance
to user query and modification.

Now, for some types, there's no way it would make sense to create an
instance directly in Python (such as, say, the "application
context"). For others, maybe you can create one in Python with the
right constructor arguments. In the first case, you could just use my
'py_bound_pod' Py::PythonExtension-derived class, constructing it around
a reference to the live some_pod instance. In the second case, you can
use one of the "owning_wrapper" templates I've written that bind
together a C++ instance and the Py::PythonExtension-derived type that
wraps it. The owning_wrapper template type derives publicly from the
Py::PythonExtension-derived type. Here's part of that facility:

==================================================
template <class Owned>
class concrete_owner_base
{
public:
	typedef Owned owned_type;

	owned_type& get_owned() throw()
	{ return m_owned; }

	const owned_type& get_owned() const throw()
	{ return m_owned; }

protected:
	concrete_owner_base()
	{}

	concrete_owner_base(const owned_type& owned)
		: m_owned( owned )
	{}

private:	// TODO: should this just be protected?
	owned_type m_owned;
};



// Assumptions:
// 1. class Base is constructible from an Owned&.

template <class Base, class Owned>
class concrete_owning_wrapper : public concrete_owner_base<Owned>,
								public Base
{
public:
	typedef Base base_type;

	concrete_owning_wrapper()
		: base_type( get_owned() )
	{}

	concrete_owning_wrapper(const owned_type& owned)
		: concrete_owner_base<owned_type>( owned ),
		  base_type( get_owned() )
	{}
};
==================================================

There's also a 'ptr_owning_wrapper' that holds a std::auto_ptr to a
heap-allocated C++ instance of the "Owned" type. By putting the
"Owned" type first in the class layout, we ensure that it lives longer
than the wrapper around it.

With all of this in place, I can create and use a wrapper around
some_pod more or less with CXX's existing, simple facilities. We can
also permit construction of a some_pod/py_bound_pod pair from Python,
using the owning_wrapper templates to unite their lifetimes. Things
get much more complicated with the nested list and map.

Next, I tried to write a Py::PythonExtension-derived class that wraps
a const std::list<>&. Wrapping a non-const reference will warrant more
methods being present, so I held off on that for now. It took a while
to get to where I am, but I still feel that I'm missing some big
points.

Take, for example, sequence_concat(). Should it be able to join with any
kind of sequence, or just another instance of our same type? What type
of sequence should it return? If we're wrapping a std::list, should we
bother creating a new std::list to underly another wrapper around it,
or should we just stuff the merged contents into a Py::List? Here, you
can see me fighting with each choice:

==================================================
Py::Object sequence_concat(const Py::Object& j)
{
    // Should we check if this is the same type as we are?
    const Py::Sequence seq( j );

    // Should we return a new instance of our own type...
/*  typedef concrete_owning_wrapper<this_type, wrapped_type> owner_type;

    std::auto_ptr<owner_type> powner( new owner_type( m_wrapped ) );

    wrapped_type& wrapped = powner->get_owned();
    wrapped.insert( wrapped.end(),
                    seq.begin(), seq.end() );

    return Py::asObject( powner );
*/

    // ... or a Py::List instance?
    Py::List listConcat( m_wrapped.size() + seq.size() );
    for ( typename wrapped_type::const_iterator
          it = m_wrapped.begin(),
          itEnd = m_wrapped.end();
          it != itEnd; ++it )
        listConcat.append( Py::make_object( *it ) );

    std::copy( seq.begin(), seq.end(),
               std::back_inserter( listConcat ) );

    return listConcat;
}
==================================================

Here, I've used some extensions to CXX to make dealing with
auto_ptr<>s safe (returning them properly through an overload of the
existing Py::asObject function) and methods like push_back() added to
class Py::List so that it works with std::back_inserter. There's also
that Py::make_object() function that I'll get to in a minute.

All of this gets even more complicated if you consider that the values
in the std::list may not be something as simple as an integer. What if
it's a type that deserves Python's "reference semantics" around it?
The elements of a list resulting from sequence_concat should be
references back to the same objects "referred to" in the source
lists. I can't figure out how to do this correctly.

Part of the problem comes from the fact that many of the "native"
Python types, as wrapped in CXX, are derived from Py::Object. You can
create them, copy them, and all the reference counting works
properly. But for Py::PythonExtension-derived types, they do not work
as simply as Py::Objects. Yes, you can create an instance and build
several Py::ExtensionObject instances around it. It's that creation of
the original instance that seems weird to me.

Consider an even simpler example: the sequence_item() method. Here's
part of my implementation:

==================================================
Py::Object sequence_item(int i)
{
    py_std_cont::seek_item_by_cached_index( m_wrapped, i,
                                            m_index, m_it );

    return Py::make_object( *m_it );
}
==================================================

It does some fancy iterator caching (with m_index and m_it) to avoid
starting at begin() every time it needs to get to an iterator by
index. The trouble is, what should it return?

Yes, we know it should return a Py::Object, presumably wrapped around
either a native Python type instance (e.g. Py::Int) or, in the more
complicated case, a Py::PythonExtension-derived instance. It works
fine when the type of the std::list is some like int than we can toss
back a Py::Int instance of.

Imagine, though, if we had std::list<some_pod> as the list type. We
have class py_bound_pod that can be constructed around a reference to
a some_pod instance. But py_bound_pod can't be returned as an instance
of Py::Object. We'd have to create a py_bound_pod on the heap, give it
to a Py::Object to own, and return that, like this:

==================================================
template <class T>
inline typename py_traits<T>::py_type make_object(const T& val)
{
    typedef typename py_traits<T>::py_type py_type;
    return Py::asObject( std::auto_ptr<py_type>( new py_type( val ) ) );
}
==================================================

I also have these py_traits structs specialized for most of the built-in
types that can return things like Py::Int from int directly:

==================================================
#define DECL_MAKEPY_BY_VAL(ctype) \
inline py_traits<ctype>::py_type make_object(ctype v) \
{ return py_traits<ctype>::py_type( v ); }


#define DECL_MAKEPY_BY_REF(ctype) \
inline py_traits<ctype>::py_type make_object(const ctype& v) \
{ return py_traits<ctype>::py_type( v ); }


DECL_MAKEPY_BY_VAL( int )
DECL_MAKEPY_BY_VAL( long )
DECL_MAKEPY_BY_VAL( float )
DECL_MAKEPY_BY_VAL( double )
DECL_MAKEPY_BY_VAL( char )
DECL_MAKEPY_BY_VAL( const char* )
DECL_MAKEPY_BY_REF( std::string )
==================================================

Granted, it's not complete, but hopefully you can see the problem I'm
trying to solve.

In the first case above, where we heap-allocate a new
Py::PythonExtension-derived wrapper around the underlying C++ type *on
each call* to sequence_item() produces a bad situation: each read of
the item creates a new Python object. Read that carefully. We'd _like_
to have each read return a new Py::Object instance pointing to the
_same_ underlying reference-counted object. Instead, we get a new
Py::Object instance pointing to a _different_ reference-counted
object, which happens to eventually point to the same wrapped C++
instance. But from Python's point of view, the following
(hypothetical) code would not behave properly:

>>> list = something.get_list_of_pods()
>>> r1 = list[0]
>>> r2 = list[0]
>>> r1 == r2
<?> - should be 1, right?
>>> r1 is r2
<?> - should be 1, right?

Each call to list[0] would create a new Py::PythonExtension-derived
instance. r1 and r2 would not be managing the same reference-counted
object. The _value_ of those objects would probably be the same since
each points to the same C++ instance under the covers, but we'd still
be violating Python's built-in mechanics.

One solution I can see is to create a parallel list that holds the
"one true wrapper" for each of the underlying list elements, and the
return Py::Object instances that point to our "one true wrapper"
instances. That gets a little more ugly to manage, and I haven't tried
that yet because I fear I may already be too far off-base.

This is a long post and it only begins to cover what's plaguing my CXX
progress. I'd be happy to post more complete code if necessary. Any
corrections and clarifications of my (erroneous) assumptions would be
most welcome. Is anyone else trying to do this sort of thing?

-- 
Steven E. Harris
Primus Knowledge Solutions, Inc.
http://www.primus.com



From barry at scottb.demon.co.uk  Sun Jun  4 14:49:49 2000
From: barry at scottb.demon.co.uk (Barry Scott)
Date: Sun, 4 Jun 2000 13:49:49 +0100
Subject: [C++-SIG] (Still) confused about wrapping C++ types with CXX
In-Reply-To: <svd7m16otl.fsf@hodge.primus.com>
Message-ID: <000001bfce23$5e7c39c0$060210ac@private>

Steve,

	Maybe this is the time to try and put a FAQ together.

	Some of what you are asking seems to come down the following
	questions:

	Q1: How does Python get its first object from the C++ application?

	A1: Starting point objects will be returned from attributes of a
	module. The attributes can be variables or functions.

	Q2: Object X exposes a list/map of object Y

	A2A: Make a copy of the C++ list/map as a Python list/map. Each
	value in the list/map will be a wrapper around a C++ object Y.

	A2B: Add methods to the wrapping of object X that allow typical
	operations on the list/map.

	A2C: Add a wrapper object that forwards the Python list/map operations
	into the list/map exposed by object X.

	Which of A2A, A2B or A3C that applies depends on the nature of the
	list/map that is exposed.

	Q3: Can I have many wrappers around a single C++ object instance?

	A3a: No if you expect the Python code to be able to compare object
	identities. Arrange that the same wrapper is return for each unique
	C++ object whose identity is critical for the applications operation.

	A3b: Yes if object identity is not important.

	Q4: How should function sequence_XXX/map_XXX be implemented?

	A4: The implementation depends on the semantics of the C++ objects
	API that is being exposed as a sequence or map. There isn't a
	generic answer.

			BArry




From skeltobc at mailandnews.com  Wed Jun  7 19:25:16 2000
From: skeltobc at mailandnews.com (Ben)
Date: Wed, 7 Jun 2000 19:25:16 +0200
Subject: [C++-SIG] HELP: wrapping with CXX, multiple inheritance issue
Message-ID: <00060719251605.16720@avalon>

Hi

I am using CXX-4.2 to wrap a c++ library, but Im having some troubles with 
multiple inheritance issues. This is what I have:

class BaseObjectWrapper
  : public PythonExtension<BaseObjectWrapper>, 
    public BaseObjectInLibrary {
public:
  BaseObjectWrapper ();
  virtual ~BaseObjectWrapper ();

  Py::Object PythonMethod1 (const Py::Tuple& args);
  Py::Object PythonMethod2 (const Py::Tuple& args);

  static void init_type (void);

  STD::string asString () const;
  virtual Py::Object repr ();
};

and

class DerivedObjectWrapper
  : public PythonExtension<DerivedObjectWrapper>, 
    public DerivedObjectInLibrary,
    public BaseObjectWrapper {
public:
  DerivedObjectWrapper ();
  virtual ~DerivedObjectWrapper ();

  Py::Object PythonMethod3 (const Py::Tuple& args);
  Py::Object PythonMethod4 (const Py::Tuple& args);

  static void init_type (void);

  STD::string asString () const;
  virtual Py::Object repr ();
};

DerivedObjectWrapper inherits BaseObjectWrapper and in the library 
DerivedObjectInLibrary inherits BaseObjectInLibrary. 

Now when I compile, I get this:

../../../cxx/CXX-4.2/Include/CXX_Extensions.h: In function `static void 
Py::Pyth
onExtension<DerivedObjectWrapper>::extension_object_deallocator(PyObject *)':
../../../cxx/CXX-4.2/Include/CXX_Extensions.h:528:   instantiated from 
`Py::Pyth
onExtension<DerivedObjectWrapper>::behaviors()'
../../../cxx/CXX-4.2/Include/CXX_Extensions.h:497:   instantiated from 
`Py::Pyth
onExtension<DerivedObjectWrapper>::type_object()'
../../../cxx/CXX-4.2/Include/CXX_Extensions.h:516:   instantiated from 
`Py::Pyth
onExtension<DerivedObjectWrapper>::PythonExtension()'
pythonwrapper.cpp:63:   instantiated from here
../../../cxx/CXX-4.2/Include/CXX_Extensions.h:673: type 
`DerivedObjectWrapper' is ambiguous bas
eclass of `PyObject'

plus a whole lot of warnings about ambiguous behaviours () and 
add_varargs_method () due I guess to references to these methods from 
PythonExtension<DerivedObjectWrapper> as well as 
PythonExtension<BaseObjectWrapper> via the BaseObjectWrapper inheritance. 
Line 63 in pythonwrapper.cpp is the point where the constructor is defined. I 
can handle the ambiguity in DerivedObjectWrapper::init_type, but I don't know 
what I should be doing with the constructor. Ive programmed some in C++ 
before finding Python but Ive never attempted to use C++'s tricky stuff. 

I could of course just lose the BaseObjectWrapper inheritance in 
DerivedObjectWrapper and rewrite the BaseObjectWrapper methods in 
DerivedObjectWrapper and other derived classes, but Im sure it's not supposed 
to work that way.

Any help here would be appreciated.

cheers
--Ben



From bscott at Ridgeway-Sys.com  Thu Jun  8 17:44:39 2000
From: bscott at Ridgeway-Sys.com (Barry Scott)
Date: Thu, 8 Jun 2000 16:44:39 +0100 
Subject: [C++-SIG] HELP: wrapping with CXX, multiple inheritance issue
Message-ID: <ABAFF012E77CD211881600A0C9B42639560FFC@RIDGEWAY>

Using the CXX 5.0 is a better base then 4.2. HOwever its not the going to
fix the inheritance.

I think the root cause of your problems is that you cannot use C++
derivation
to create related Python objects.

		BArry



-----Original Message-----
From: Ben [mailto:skeltobc at mailandnews.com]
Sent: 07 June 2000 18:25
To: c++-sig at python.org
Subject: [C++-SIG] HELP: wrapping with CXX, multiple inheritance issue


Hi

I am using CXX-4.2 to wrap a c++ library, but Im having some troubles with 
multiple inheritance issues. This is what I have:

class BaseObjectWrapper
  : public PythonExtension<BaseObjectWrapper>, 
    public BaseObjectInLibrary {
public:
  BaseObjectWrapper ();
  virtual ~BaseObjectWrapper ();

  Py::Object PythonMethod1 (const Py::Tuple& args);
  Py::Object PythonMethod2 (const Py::Tuple& args);

  static void init_type (void);

  STD::string asString () const;
  virtual Py::Object repr ();
};

and

class DerivedObjectWrapper
  : public PythonExtension<DerivedObjectWrapper>, 
    public DerivedObjectInLibrary,
    public BaseObjectWrapper {
public:
  DerivedObjectWrapper ();
  virtual ~DerivedObjectWrapper ();

  Py::Object PythonMethod3 (const Py::Tuple& args);
  Py::Object PythonMethod4 (const Py::Tuple& args);

  static void init_type (void);

  STD::string asString () const;
  virtual Py::Object repr ();
};

DerivedObjectWrapper inherits BaseObjectWrapper and in the library 
DerivedObjectInLibrary inherits BaseObjectInLibrary. 

Now when I compile, I get this:

../../../cxx/CXX-4.2/Include/CXX_Extensions.h: In function `static void 
Py::Pyth
onExtension<DerivedObjectWrapper>::extension_object_deallocator(PyObject
*)':
../../../cxx/CXX-4.2/Include/CXX_Extensions.h:528:   instantiated from 
`Py::Pyth
onExtension<DerivedObjectWrapper>::behaviors()'
../../../cxx/CXX-4.2/Include/CXX_Extensions.h:497:   instantiated from 
`Py::Pyth
onExtension<DerivedObjectWrapper>::type_object()'
../../../cxx/CXX-4.2/Include/CXX_Extensions.h:516:   instantiated from 
`Py::Pyth
onExtension<DerivedObjectWrapper>::PythonExtension()'
pythonwrapper.cpp:63:   instantiated from here
../../../cxx/CXX-4.2/Include/CXX_Extensions.h:673: type 
`DerivedObjectWrapper' is ambiguous bas
eclass of `PyObject'

plus a whole lot of warnings about ambiguous behaviours () and 
add_varargs_method () due I guess to references to these methods from 
PythonExtension<DerivedObjectWrapper> as well as 
PythonExtension<BaseObjectWrapper> via the BaseObjectWrapper inheritance. 
Line 63 in pythonwrapper.cpp is the point where the constructor is defined.
I 
can handle the ambiguity in DerivedObjectWrapper::init_type, but I don't
know 
what I should be doing with the constructor. Ive programmed some in C++ 
before finding Python but Ive never attempted to use C++'s tricky stuff. 

I could of course just lose the BaseObjectWrapper inheritance in 
DerivedObjectWrapper and rewrite the BaseObjectWrapper methods in 
DerivedObjectWrapper and other derived classes, but Im sure it's not
supposed 
to work that way.

Any help here would be appreciated.

cheers
--Ben

_______________________________________________
C++-SIG maillist  -  C++-SIG at python.org
http://www.python.org/mailman/listinfo/c++-sig