From furnish at laura.llnl.gov Thu Feb 13 00:36:55 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Wed, 12 Feb 1997 15:36:55 -0800 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status Message-ID: <199702122340.SAA19351@python.org> Greetings to all again. This message is being reposted because a configuration probelm with the list was uncovered. Please excuse. This message is to report on the current status of the work being done on Python/C++ integration in my research group at LLNL. There are surely others working in this general area, I hope you will all post a summary note of your current situation so we can all take stock, and benefit from having our perspectives widened. To get things rolling, I will provide some comment directly on the points listed in the SIG charter, and them probably ramble some more after that as well. > ------------------------------------------------------------------- > C++-SIG, SIG for Development of a C++ Binding to Python > > This list exists in order to discuss the design and development of a > C++ binding for the compiled interface to Python, and associated > classes and tools for the construction of Python extension modules in > C++. > > Major issues to be discussed/resolved, include, but are not limited > to: > > 1) Autoconf support for enabling C++ support in Python. This must be > managed in a way which does not change the C API, or the behavior > of Python if not configured for C++ support, in any (observable?) > way. In other words, conventional C extension modules must > continue to work, whether Python is configured with C++ support or > not. One of the first things which has to be faced is how to build a Python which can support C++. On most systems, this means in particular, that "main" must be compiled with the C++ compiler. Moreover, lacking platform ABI's for C++, python must be built with a particular C++ compiler--the one you will use for your extension work. You can't mix and match .o's from C++ compilers the way you (often) can with .o's from C compilers. To cope with this, I modified our configure to support a new option, --with-cxx[=compiler_name] If you just say, for instance, ./configure --with-cxx it will find whatever C++ compiler the autoconf macro comes up with, which may or may not be the one you want. If it is not the one you want, then add the "=" part. For instance, if you are using the Kuck and Associates C++ compiler (for instance), you would say: ./configure --with-cxx=KCC This will then produce a makefile setup which will compile Python and link it in such a way that the main is compiled with C++, and the link is handled by C++. It is important to understand that the link must be done by C++, since not only do we need to get global constructors run (the usual and fairly well appreciated excuse, for which there exist some platform-specific workarounds), but also because we will need to let the compiler do template closure, for which there is really no substitute for just letting the compiler do the job. While I was at it, I added a --with-debug flag to configure. I observed that there was a way to get approximately this effect using variable specification at make time under Guido's original autoconf stuff, but it was not really adequate to handle the case of an executable which is built with multiple compilers, since they may not all take the same flags to turn on debugging. For example, most compilers take -g, but KCC takes "+K0". If you configure - --with-debug, then you get debugging support in both C and C++ compilations. It may be necessary to tune this autoconf stuff for popular compilers. Anyway, the result of this is that autoconf now allows building a python, either with or without C++ support. Without it, you get exactly the python you would've gotten in the first place. With it, you get a better python, one that can support C++ extension modules. I should add at this point that in our early efforts, we also modified portions of the build to support the compilation of additional parts of the python core with C++ because of the desire for 5) below. However, we backed that out, and are using an extension module for those capabilities instead. More on that later. > 2) Type safety in the Python API. Not a tremendous amount of work in this direction yet. > 3) Introducing C++ classes for the various major Python abstractions, > such as dictionaries, lists, tuples, modules, etc. PyXXX_YYY(...) ??? How about instead, XXX v; v->YYY(...); Essentially we are wrapping the Python API with a set of objects which will have methods which correspond to the various portions of the C API. This will generally eliminate the first PyObject * argument to most of these C API functions, since they can be identified immediately as the "this" object, and the API call is most naturally considered a method invocation on that object. Our goal is to make Python really look this way to the C++ programmer. Planned classes at this time include: PyDict PyList PyTuple and possible also PyArgs, although this is still under discussion since arglists are always tuples now. I am personally inclined to keep PyArgs as a seperate class (perhaps derived from PyTuple, not sure), just for the sake of specficity in callback function signatures. We have not set down and specifically carved up the entire Python C API function universe, but we expect that the above classes will get us well down the road. We would welcome suggestions about what else is needed. > 4) Providing semantics for these classes which are natural both to the > Python programmer, as well as the C++ programmer. For example, the > PyDict class should have an operator[], and should also support STL > style iterators. Many issues of this form. This is just starting, nothing quite report-worthy yet. > 5) Method invocation. How to make the invocation of C++ member > functions as natural and/or painless as possible. Our use of Python involves providing a script interface to object oriented class libraries written in C++. Integration/wrapper technologies which are geared toward wrapping API's as Python modules, are not overly useful to us, since we for the most part, do not even have global functions in our compiled subsystems. We have classes/objects. Our Python extension modules, coded in C++, need to be able to reflect this structure. Building classes which wrap the real ones, but have Python-esque signatures, seems to be one productive way to procede. However, Python can only register callbacks for global C functions. So then you wind up writing C style global functions which call C++ member functions which call the C++ services in the ultimate compiled assets. This is clumsy and we would like to avoid the extra layer if possible. To accomplish this, we have built a C++ extension module which provides the capability to invoke C++ member functions directly from Python. This is accomplished by mimicking the "function object" support in Python, with a new "member function object" type, which has the internal data necessary to allow a C++ method invocation. The Py_FindMethod() function has been succeeded by a new getattr assister named FindMemberFunc(). This function builds a C++ member function object corresponding to the requested member function for a C++ python extension class. When Python calls this member function object, a trampoline function is called which extracts the necessary data from member function object, which allows it to invoke the C++ member function on the object. This eliminates the need for the self parameter in the call back, since it is replaced by C++'s implicit "this". So, you derive your shadow object from PyObject, and this allows us to abolish the self parameter. The args all come in as a PyArgs variable, which you can pick apart with the usual ParseTuple business. > 6) Error handling. How to irradicate NULL checking in favor of C++ > style exceptions. Finally, our C++ wrapper over the Python C API allows the abolition of all NULL checking. Instead, Python exceptions raised by Python C API functions are converted directly into C++ exceptions, which begin bouncing up the call stack until they are handled by a matching catch clause. If they are not caught at all by the C++ python extension module, then they are trapped by the trampoline described in 5) which called the member function in the first place. At this point they are returned to the python core just as they would've been if you'd done all that hideous NULL checking in the first place. Complete with python traceback. The result of this is that you can now write very straightforward python extension code in C++. If you /want/ to do speculative calls, in which you are prepared to test for and cope with the results of exceptions in the Python C API, then you merely wrap such code in a try/catch clause. If you are /not/ willing to do such checking (the usual case), then you can simply proceed along your merry way, writing code, calling the Python C API (which you will problaby do throug the wrapper objects), ignoring return codes, confident that any real errors which result will propagate immediately back to the call site and be returned as the familiar stack trace, just as they should. We feel this makes for much more lucid/robust coding. Note that the trampoline not only catches the exceptions thrown by the Python C API wrappers, but it will also catch /any/ C++ exception which may percolate out of a C++ python extension module. Thus, any failure (if registered as a C++ exception) in C++ extension code, will be converted to a python exception with accompanying stack trace. This represents a dramatic improvement in the state of affairs of Python steering of computational engines, from our perspective. > ------------------------------------------------------------------- To see how some of this fits together, here is the cxxtstmodule.cc file which we are using to exercise these features. //----------------------------------*-C++-*----------------------------------// // cxxtstmodule.cc //---------------------------------------------------------------------------// #include #include "PythonX.hh" using namespace Py; static char cxxtst_alive__doc__[] = "alive(): Verify that cxxtst is alive."; static PyObject * cxxtst_alive( PyObject *self, PyArgs *args ) { args->ParseTuple(); cout << "Yes, the C++ extension binding test module is alive!\n" << flush; Py_INCREF(Py_None); return Py_None; } static char cxxtst_f1__doc__[] = "f1(): Function taking one arg."; static PyObject * cxxtst_f1( PyObject *self, PyArgs *args ) { int i; args->ParseTuple( i ); cout << "Yes, the user passed a single int argument: " << i << endl; Py_INCREF(Py_None); return Py_None; } static char cxxtst_toss_cookies__doc__[] = "toss_cookies(): Function throwing a C++ exception."; static PyObject * cxxtst_toss_cookies( PyObject *self, PyArgs *args ) { cout << "toss_cookies is heaving.\n" << flush; throw "Oops, better clean up the mess!"; Py_INCREF(Py_None); return Py_None; } //---------------------------------------------------------------------------// // Now stuff for a type Foo. class Foo : public PyObject { public: Foo(); PyObject *munge( PyArgs *args ); PyObject *crash( PyArgs *args ); }; static void Foo_dealloc( Foo *pf ); static PyObject *Foo_getattr( Foo *pf, char *name ); statichere PyTypeObject Footype = { PyObject_HEAD_INIT(&PyType_Type) 0, "Foo", sizeof(Foo), 0, (destructor) Foo_dealloc, /*tp_dealloc*/ 0, /*tp_print*/ (getattrfunc) Foo_getattr, /*tp_getattr*/ 0, /*tp_setattr*/ 0, /*tp_compare*/ 0, /*tp_repr*/ 0, /*tp_as_number*/ 0, /*tp_as_sequence*/ 0, /*tp_as_mapping*/ 0, /*tp_hash*/ }; #define is_Fooobject(op) ((op)->ob_type == &Footype) Foo::Foo() { ob_type = &Footype; } PyObject *Foo::munge( PyArgs *args ) { cout << "In Foo::munge\n" << flush; Py_INCREF( Py_None ); return Py_None; } PyObject *Foo::crash( PyArgs *args ) { cout << "In Foo::crash\n" << flush; throw "flaming oblivion"; Py_INCREF( Py_None ); return Py_None; } X_MethodDef XFoo_methods[] = { { "munge", Foo::munge, 1, "do some work" }, { "crash", Foo::crash, 1, "die horribly" }, { NULL, NULL, NULL, NULL } }; //---------------------------------------------------------------------------// // "Methods" of the C extension type. static void Foo_dealloc( Foo *pf ) { delete pf; } /* THE STANDARD GETATTR FOR NAMED METHODS */ static PyObject *Foo_getattr( Foo *pf, char *name ) { return FindMemberFunc( XFoo_methods, pf, name ); } PyObject *new_Foo( PyObject *self, PyArgs *args ) { return new Foo; } Instantiate_helpers(Foo) //---------------------------------------------------------------------------// static char cxxtst_module_documentation[] = "Python extension module for testing the C++ bindings for extensions."; // List of methods defined in the module static struct PyMethodDef cxxtst_methods[] = { {"alive", (PyCFunction) cxxtst_alive, 1, cxxtst_alive__doc__}, {"f1", (PyCFunction) cxxtst_f1, 1, cxxtst_f1__doc__}, {"toss_cookies", (PyCFunction) cxxtst_toss_cookies, 1, cxxtst_toss_cookies__doc__}, {"new_Foo", (PyCFunction) new_Foo, 1, }, {NULL, NULL, NULL, NULL} /* sentinel */ }; extern "C" void initcxxtst() { // Create the module and add the functions. Module *m = XInitmodule4( "cxxtst", cxxtst_methods, cxxtst_module_documentation, NULL, PYTHON_API_VERSION ); // Check for errors if (PyErr_Occurred()) Py_FatalError("can't initialize module cxxtst"); } //---------------------------------------------------------------------------// // end of cxxtstmodule.cc //---------------------------------------------------------------------------// Here is what I get when I try various possible operations with it: mousey[3] ./python >>> import cxxtst >>> cxxtst.alive() Yes, the C++ extension binding test module is alive! >>> cxxtst.alive(-1) Traceback (innermost last): File "", line 1, in ? TypeError: function requires exactly 0 arguments; 1 given >>> f = cxxtst.new_Foo() >>> f.munge() In Foo::munge >>> f.crash() In Foo::crash Traceback (innermost last): File "", line 1, in ? RuntimeError: a C++ exception has occurred: flaming oblivion >>> To understand how this works, you need to see the invocation trampolines. The one for normal C++ global functions is: static PyObject *cxx_trampoline_va( cxxmethodobject *self, PyObject *args ) { try { return (*self->m_ml->ml_meth)( self->m_self, args ); } catch( PyException& pyx ) { // This happens when the Python C API has already set an exception // condition, which is indicated by a return value of NULL. return NULL; } catch( const char *msg ) { char buf[ 1024 ]; sprintf( buf, "a C++ exception has occurred: %s", msg ); PyErr_SetString( PyExc_RuntimeError, buf ); return NULL; } catch(...) { PyErr_SetString( PyExc_RuntimeError, "an unrecognized C++ exception has occurred." ); return NULL; } } And the one for C++ member functions is: namespace Py { PyObject *cxx_trampoline_mbrfnc_va( cxxmemberfuncobject *self, PyObject *args ) { PyObject *s = self->m_self; PyArgs *arglist = static_cast( args ); try { return (s->*(self->m_ml->ml_meth))( arglist ); } catch( PyException& pyx ) { // This happens when the Python C API has already set an exception // condition, which is indicated by a return value of NULL. return NULL; } catch( const char *msg ) { char buf[ 1024 ]; sprintf( buf, "a C++ exception has occurred: %s", msg ); PyErr_SetString( PyExc_RuntimeError, buf ); return NULL; } catch(...) { PyErr_SetString( PyExc_RuntimeError, "an unrecognized C++ exception has occurred." ); return NULL; } } You can compare these to the functions in the Python core to see the correspondence. Basically we just use a try/catch wrapper clause, and provide special handling for method invocation using a method pointer. Another thing worth calling attention to is the code: X_MethodDef XFoo_methods[] = { { "munge", Foo::munge, 1, "do some work" }, { "crash", Foo::crash, 1, "die horribly" }, { NULL, NULL, NULL, NULL } }; static PyObject *Foo_getattr( Foo *pf, char *name ) { return FindMemberFunc( XFoo_methods, pf, name ); } This is the replacement for Py_FindMethod(). From this you can see that FindMemberFunc is templated, so that it is possible to construct the method search engine for each type of Python extension object, which avoids ugly casts. Final comments. By this point, people should be getting the pretty clear image that these extensions exercise the C++ language at a level very close to the draft standard. This is without apology. We are coding under the assumption that all of these are true: compiler has "real" template support compiler has member templates compiler has exceptions compiler has rtti compiler has namespaces This is the C++ language as we know it, and we are not buckling under on making a quality interface on account of compiler vendors who don't support the language as the C++ committee defines it. This may well mean that certain C++ compilers in the market place will not compile this code. If you have such a compiler, you will need to wait until your compiler is upgraded to a level conformant with the language spec, before you will be able to use this. Availability of the LLNL C++ patch set: The code described in this note will be available to others. We have not made clear release plans/schedules yet. It would be useful to get a reading on how much interest there is in this. Certainly it is an ongoing development effort, with all the cautionary statements about instability that go along with that. Nevertheless, we are highly motivated to get it into a stable form for our own purposes, and are prepared to cut occasional patch sets for external use. Cheers to all. Let's hear what others are doing. - -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From da at maigret.cog.brown.edu Thu Feb 13 02:11:19 1997 From: da at maigret.cog.brown.edu (David Ascher) Date: Wed, 12 Feb 1997 20:11:19 -0500 (EST) Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <199702122340.SAA19351@python.org> from "Geoffrey Furnish" at Feb 12, 97 03:36:55 pm Message-ID: <199702130111.UAA03130@maigret> Hi all. Geoff said: > Anyway, the result of this is that autoconf now allows building a > python, either with or without C++ support. Without it, you get > exactly the python you would've gotten in the first place. With it, > you get a better python, one that can support C++ extension modules. Just a question: how do you deal with frozenmain.c? Do you treat it as a C++ file? I assume you've also modified the .c files to do the appropriate extern "C" {...} stuff... > To accomplish this, we have built a C++ extension module which > provides the capability to invoke C++ member functions directly from > Python. This is accomplished by mimicking the "function object" > support in Python, with a new "member function object" type, which has > the internal data necessary to allow a C++ method invocation. The > Py_FindMethod() function has been succeeded by a new getattr assister > named FindMemberFunc(). This function builds a C++ member function > object corresponding to the requested member function for a C++ python > extension class. When Python calls this member function object, a > trampoline function is called which extracts the necessary data from > member function object, which allows it to invoke the C++ member > function on the object. Cool. > [description of argument tuplification to callbacks which I didn't > mean to delete] Is it possible to have the argument detuplification and parsing automated when so desired? So that if I specify a callback as expecting 2 ints, I can define those in the method definition and the 'system' will deal with the ArgTuple handling? It seems it would be a nice feature, if it's possible [this is getting into areas of C++ that I don't know well]. > Finally, our C++ wrapper over the Python C API allows the abolition of > all NULL checking. Instead, Python exceptions raised by Python C API > functions are converted directly into C++ exceptions, which begin Very nice. > #include > > #include "PythonX.hh" This maybe too late, but 'PythonX' makes me think of Python/ActiveX, which is a completely different system... > This is the C++ language as we know it, and we are not buckling under > on making a quality interface on account of compiler vendors who don't > support the language as the C++ committee defines it. This may well > mean that certain C++ compilers in the market place will not compile > this code. If you have such a compiler, you will need to wait until > your compiler is upgraded to a level conformant with the language > spec, before you will be able to use this. This is not at all a criticism, but I'm curious: does the latest g++ qualify? I think this will have at least some impact on the public reaction to your system. > The code described in this note will be available to others. We have > not made clear release plans/schedules yet. It would be useful to > get a reading on how much interest there is in this. Certainly it is > an ongoing development effort, with all the cautionary statements > about instability that go along with that. Nevertheless, we are > highly motivated to get it into a stable form for our own purposes, > and are prepared to cut occasional patch sets for external use. What are your thoughts regarding eventual integration in the main source distribution? It strikes me that such an integration would be a very worthwhile effort, but I realize it is not one without compromises or headaches. This sounds really impressive -- congratulations to all involved. --david ascher _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Thu Feb 13 18:34:03 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Thu, 13 Feb 1997 09:34:03 -0800 Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status In-Reply-To: <199702130111.UAA03130@maigret> References: <199702122340.SAA19351@python.org> <199702130111.UAA03130@maigret> Message-ID: <199702131737.MAA24664@python.org> David Ascher writes: > Hi all. > > Geoff said: > > Anyway, the result of this is that autoconf now allows building a > > python, either with or without C++ support. Without it, you get > > exactly the python you would've gotten in the first place. With it, > > you get a better python, one that can support C++ extension modules. > > Just a question: how do you deal with frozenmain.c? Do you treat it > as a C++ file? Uhhh. Never heard of it. This is an excellent example of why we need to do this in a public forum. Thanks for calling it to my attention. Suggestions welcome. > I assume you've also modified the .c files to do the appropriate > extern "C" {...} stuff... I just copied main.c to cxxmain.cc, and compile that with C++ and link it in. The makefiles know which one to use, based on whether or not it was configured --with-cxx. > > [description of argument tuplification to callbacks which I didn't > > mean to delete] > > Is it possible to have the argument detuplification and parsing > automated when so desired? So that if I specify a callback as expecting > 2 ints, I can define those in the method definition and the 'system' > will deal with the ArgTuple handling? It seems it would be a nice > feature, if it's possible [this is getting into areas of C++ that I > don't know well]. Ohhhhhhh yes. Yes yes yes. I am working on a tool I call the "pythalyzer" (nomenclature would be obvious only to those I've worked with over the years, who have been subjected to an endless stream of xxxalyzers for various automatic code generation purposes). It is not really ready for public review yet, but the basic idea is that it can generate these shadow classes for you, by constructing a Python class (by which I mean, a C++ class for the python extension module) which directly shadows a particular C++ class (the compiled asset you are trying to propagate to the Python script level). This wrapper class will have one method for each method in the real class. The wrapper will have the characteristic signature PyObject *::( PyArgs *args ); (Recall from yesterday's post that "self" has been expunged, since it is trivially identified as "this" inside C++ classes called directly from Python). The action of this generated method will be to suck the arguments out of the argument list tuple, and then call the method on the real C++ class. This is all done with dufus cut and paste, so it is up to the compiler to figure out the types of everything. (This is good, b/c the compiler is never wrong, so this plan is not subject to human error). This is another area where our C++ interface will de-C-ize the Python API. Instead of calling ParseTuple with a format string and gobs of pointers, you'll just call it directly with the variables you want filled in. The template system, using the C++ "traits" technique, will construct the format string for you, and populate the arguments. This should make ParseTuple foolproof, and imune to the wide variety of catistrophic errors which acrue to all systems which use printf style format string semantics. > > #include "PythonX.hh" > > This maybe too late, but 'PythonX' makes me think of Python/ActiveX, which > is a completely different system... This is under discussion here. Suggestions welcome. Python++.hh is another posibility, but I'm not sure how portable that is to systems with pathetic filesystems (biting tounge). Ultimately, this stuff could coneivably be made available directly through the usual Python.h, but I think that is way down the road. Anyway, "nothing is too late". The reason we are discussing this here is specifically so that we can find out what people think, and we are prepared to act on good suggestions (timescale unspecified of course). > > This is the C++ language as we know it, and we are not buckling under > > on making a quality interface on account of compiler vendors who don't > > support the language as the C++ committee defines it. This may well > > mean that certain C++ compilers in the market place will not compile > > this code. If you have such a compiler, you will need to wait until > > your compiler is upgraded to a level conformant with the language > > spec, before you will be able to use this. > > This is not at all a criticism, but I'm curious: does the latest g++ > qualify? I think this will have at least some impact on the public > reaction to your system. 2.7.2 is certainly nowhere near adequate. I don't think 2.8.0 is out yet, but early indications I have received make me fear it will also be inadequate, despite being a dramatic improvement over 2.7.2. We are aware this is a short term liability. Be we do believe it is a /short/ term liability, and in our group here at LLNL, we simply do not work with compilers that do not conform to a recent edition of the draft. It will be a great day indeed, when g++ is ANSI C++ compatible, and we look forward to that day, but we are not prepared to wait idly for that day. > > The code described in this note will be available to others. We have > > not made clear release plans/schedules yet. It would be useful to > > get a reading on how much interest there is in this. Certainly it is > > an ongoing development effort, with all the cautionary statements > > about instability that go along with that. Nevertheless, we are > > highly motivated to get it into a stable form for our own purposes, > > and are prepared to cut occasional patch sets for external use. > > What are your thoughts regarding eventual integration in the main source > distribution? It strikes me that such an integration would be a very > worthwhile effort, but I realize it is not one without compromises or > headaches. We have in fact already paid some of this "cost of compromise". My original design for the C++ trampoline was a direct modification to the python core. In my opinion, this is where it fits best. But there was significant concern in our group that such an approach was too invasive for the realistic expectation of being accepted into the Python official release, so I backed it out to extension status. We would be very interested in having this work accepted into the Python standard distribution, in the full course of time. In order for that to be realistic, we realize the code needs to be reviewed by a significantly wider audience than just our little research group, which is why I lobbied for this SIG. If it becomes clear through this forum that our work is of interest to others, we will cut releases of our patch set for public use. We will consider suggestions made by others. And if/when this system achieves a reasonable degree of stability and maturity, we would be happy to see it absorbed into the Python official distribution. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From neal at ctd.comsat.com Thu Feb 13 18:57:46 1997 From: neal at ctd.comsat.com (Neal Becker) Date: 13 Feb 1997 12:57:46 -0500 Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status In-Reply-To: Geoffrey Furnish's message of Thu, 13 Feb 1997 09:34:03 -0800 References: <199702122340.SAA19351@python.org> <199702130111.UAA03130@maigret> <199702131737.MAA24664@python.org> Message-ID: Out of curiosity, what c++ compiler are you using, that meets the requirements? _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From da at maigret.cog.brown.edu Thu Feb 13 19:17:10 1997 From: da at maigret.cog.brown.edu (David Ascher) Date: Thu, 13 Feb 1997 13:17:10 -0500 (EST) Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status In-Reply-To: <199702131737.MAA24664@python.org> Message-ID: On Thu, 13 Feb 1997, Geoffrey Furnish wrote: > > Just a question: how do you deal with frozenmain.c? Do you treat it > > as a C++ file? > > Uhhh. Never heard of it. This is an excellent example of why we need > to do this in a public forum. Thanks for calling it to my attention. > Suggestions welcome. As you may have figured out by now, it's the main that's used when freezing objects. I suspect it might make sense to adjust freeze to allow a --with-cxx flag as well... Pythanalyzer sounds good too. > > > #include "PythonX.hh" > > > > This maybe too late, but 'PythonX' makes me think of Python/ActiveX, which > > is a completely different system... > > This is under discussion here. Suggestions welcome. Python++.hh is > another posibility, but I'm not sure how portable that is to systems > with pathetic filesystems (biting tounge). I think it's funny that you are ruthless on ANSI C++ compatibility but not decent filename support from the FS. =) > 2.7.2 is certainly nowhere near adequate. I don't think 2.8.0 is out > yet, but early indications I have received make me fear it will also > be inadequate, despite being a dramatic improvement over 2.7.2. > > We are aware this is a short term liability. Be we do believe it is a > /short/ term liability, and in our group here at LLNL, we simply do > not work with compilers that do not conform to a recent edition of the > draft. It will be a great day indeed, when g++ is ANSI C++ > compatible, and we look forward to that day, but we are not prepared > to wait idly for that day. I'd be interested (as Neal Becker asked) in what compiler you're using, and maybe it'd be worthwhile maintaining a list of which compilers would be acceptable. For example, which of of the SGI C++ compilers fit the bill, if any? > We would be very interested in having this work accepted into the > Python standard distribution, in the full course of time. In order > for that to be realistic, we realize the code needs to be reviewed by > a significantly wider audience than just our little research group, > which is why I lobbied for this SIG. If it becomes clear through this > forum that our work is of interest to others, we will cut releases of > our patch set for public use. We will consider suggestions made by > others. And if/when this system achieves a reasonable degree of > stability and maturity, we would be happy to see it absorbed into the > Python official distribution. Guido will tell us if he's subscribed, but I got the definite impression at the latest SPAM that he is interested in dealing with C++ in the core as long as it's a complete and good solution. _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Thu Feb 13 19:20:01 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Thu, 13 Feb 1997 10:20:01 -0800 Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status In-Reply-To: References: <199702122340.SAA19351@python.org> <199702130111.UAA03130@maigret> <199702131737.MAA24664@python.org> Message-ID: <199702131823.NAA24887@python.org> Neal Becker writes: > Out of curiosity, what c++ compiler are you using, that meets the > requirements? One popular compiler which is available on many hosts, is Kuck and Associates compiler, "KAI C++". See http://www.kai.com. The Portland Group, Inc, also sells a compiler which is supposedly "up to snuff". -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From bwarsaw at CNRI.Reston.Va.US Thu Feb 13 21:03:39 1997 From: bwarsaw at CNRI.Reston.Va.US (Barry A. Warsaw) Date: Thu, 13 Feb 1997 15:03:39 -0500 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status References: <199702122340.SAA19351@python.org> Message-ID: <199702132003.PAA00824@anthem.CNRI.Reston.Va.US> First an aside: I have not tracked C++ developments very closely since coming here to CNRI about 2 1/2 years ago, so I'm only dimly aware of details of the new features like RTTI, exceptions, and namespaces. I used to have one of the earliest editions of the ARM. Would you recommend getting a new edition, or is there a better reference manual I can get to bone up on these features? All in all, not doing any real C++ work these days, I still think your approach looks really good. I like the natural feel this would give to a C++er. Please excuse the few pretty trivial comments, suggestions, and questions. >>>>> "GF" == Geoffrey Furnish writes: > 3) Introducing C++ classes for the various major Python abstractions, > such as dictionaries, lists, tuples, modules, etc. GF> Python really look this way to the C++ programmer. Planned GF> classes at this time include: GF> PyDict GF> PyList GF> PyTuple GF> and possible also PyArgs, I'd only suggest adding the word `Class' on the end here, (or maybe `Wrapper') e.g. PyDictClass. Although `PyDict' may never conflict with other names in the C API, there are symbols like `PyDict_Type'. It seems like it could cause confusion, for example, what if you passed a PyDict object to PyDict_Type? That would seem to me a natural thing to want to do given the naming. > 6) Error handling. How to irradicate NULL checking in favor of C++ > style exceptions. GF> Finally, our C++ wrapper over the Python C API allows the GF> abolition of all NULL checking. This is very cool. [code deleted] On the C++ coding styles. I used to code C++ very similar to the style you seem to use. This is at odds with the standard Python C coding style. If this stuff ever makes it into the core (and I hope something like it does), it will make life easier if the stuff to be integrated looks like the core C style, where appropriate of course. [If you're an Emacs user, the latest cc-mode has a "python" style that gets you close]. Maybe Guido can post some pointers to his style guide. Maybe he'd just tell me its too trivial to matter right now. Maybe I've just been mucking about in the Python core too much. :-) GF> By this point, people should be getting the pretty clear image GF> that these extensions exercise the C++ language at a level GF> very close to the draft standard. I respect this decision, and actually hope you can get away with it, especially since it makes the C++ API that much cleaner. GF> human error). This is another area where our C++ interface GF> will de-C-ize the Python API. Instead of calling ParseTuple GF> with a format string and gobs of pointers, you'll just call it GF> directly with the variables you want filled in. The template GF> system, using the C++ "traits" technique, will construct the GF> format string for you, and populate the argumenfs. Could you give me a quick primer on what the "traits" technique is? This is definitely something new to me. -Barry _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From donb at zippy.boston.sgi.com Thu Feb 13 21:21:43 1997 From: donb at zippy.boston.sgi.com (Donald Beaudry) Date: Thu, 13 Feb 1997 15:21:43 -0500 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status References: <199702122340.SAA19351@python.org> <199702132003.PAA00824@anthem.CNRI.Reston.Va.US> Message-ID: <199702132021.PAA08910@zippy.boston.sgi.com> "Barry A. Warsaw" wrote, > Could you give me a quick primer on what the "traits" technique is? > This is definitely something new to me. Check out , but I'm real interested in hearing about how this is going to get applied to the C++/Python issue. --Don _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From johann at physics.berkeley.edu Thu Feb 13 22:04:44 1997 From: johann at physics.berkeley.edu (Johann Hibschman) Date: Thu, 13 Feb 1997 13:04:44 -0800 (PST) Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status In-Reply-To: <199702131737.MAA24664@python.org> Message-ID: On Thu, 13 Feb 1997, Geoffrey Furnish wrote: > David Ascher writes: > > This is not at all a criticism, but I'm curious: does the latest g++ > > qualify? I think this will have at least some impact on the public > > reaction to your system. > > 2.7.2 is certainly nowhere near adequate. I don't think 2.8.0 is out > yet, but early indications I have received make me fear it will also > be inadequate, despite being a dramatic improvement over 2.7.2. > > We are aware this is a short term liability. Be we do believe it is a > /short/ term liability, and in our group here at LLNL, we simply do > not work with compilers that do not conform to a recent edition of the > draft. It will be a great day indeed, when g++ is ANSI C++ > compatible, and we look forward to that day, but we are not prepared > to wait idly for that day. This one *is* meant to be a criticism. By requiring features that are not in common distribution, you're cutting out a sizeable chunk of the user base. LLNL may have pots of money to buy the latest optimizing C++ compiler, but I am stuck either using the Sun CC compiler or g++. I don't think that either is in full compliance with the ANSI draft standard, although I'm not sure about the latest version of Sun's offering. I don't think that I am that atypical of academia, and I don't want to wait a year or two for g++ to get up to par. If this is a mainstream effort to get C++ support for python, it should be usable by mainstream compilers. If this is really just a forum to perfect your idealized C++ implementation, then there's probably enough demand for a separate "lowly C++" SIG for everyone else. - Johann --- Johann Hibschman johann at physics.berkeley.edu _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From bwarsaw at CNRI.Reston.Va.US Thu Feb 13 22:21:06 1997 From: bwarsaw at CNRI.Reston.Va.US (Barry A. Warsaw) Date: Thu, 13 Feb 1997 16:21:06 -0500 Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status References: <199702131737.MAA24664@python.org> Message-ID: <199702132121.QAA00933@anthem.CNRI.Reston.Va.US> >>>>> "JH" == Johann Hibschman writes: JH> If this is really just a forum to perfect your idealized C++ JH> implementation, then there's probably enough demand for a JH> separate "lowly C++" SIG for everyone else. Why a separate SIG? This one is perfectly suitable to discuss such issues, IMHO. Someone should do a survey of the commonly available compilers for the various platforms (don't forget non-Unix platforms!) and report on the state-of-the-art. Maybe even contact the vendors/maintainers and see if you can get some estimates on when such features would be supported (yeah right :-). -Barry _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From dubois1 at llnl.gov Thu Feb 13 22:27:20 1997 From: dubois1 at llnl.gov (Paul F. Dubois) Date: Thu, 13 Feb 1997 13:27:20 -0800 Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status Message-ID: <9702132127.AA21632@icf.llnl.gov.llnl.gov> ---------- > From: Johann Hibschman > To: c++-sig at python.org > Subject: Re: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status > Date: Thursday, February 13, 1997 1:04 PM > > This one *is* meant to be a criticism. By requiring features that are not > in common distribution, you're cutting out a sizeable chunk of the user > base. LLNL may have pots of money to buy the latest optimizing C++ > compiler, but I am stuck either using the Sun CC compiler or g++. I don't > think that either is in full compliance with the ANSI draft standard, > although I'm not sure about the latest version of Sun's offering. > > I don't think that I am that atypical of academia, and I don't want to > wait a year or two for g++ to get up to par. If this is a mainstream > effort to get C++ support for python, it should be usable by mainstream > compilers. If this is really just a forum to perfect your idealized C++ > implementation, then there's probably enough demand for a separate "lowly > C++" SIG for everyone else. > > - Johann > > --- > Johann Hibschman > johann at physics.berkeley.edu This is not a decision we made lightly. We don't think the widespread availability of compilers that do meet the C++ standard is all that far away. Meanwhile, it would be pound -foolish to expend the time of our personnel to write convoluted work-arounds rather than buy a modestly-priced compiler that DOES work. By the way, the issue for this SIG is actually not the optimization, it is simply the use of the full language as specified in the standard. We care about the optimization when we have our C++ hats on, but for the purpose of the Python interface to our C++ speed is not often something to think about. If our experience with the matrix-SIG is any experience, we will argue about the details of this longer than it will take for the compilers to be out. _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Thu Feb 13 22:36:04 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Thu, 13 Feb 1997 13:36:04 -0800 Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status In-Reply-To: References: <199702131737.MAA24664@python.org> Message-ID: <199702132139.QAA26174@python.org> Johann Hibschman writes: > On Thu, 13 Feb 1997, Geoffrey Furnish wrote: > > > David Ascher writes: > > > > This is not at all a criticism, but I'm curious: does the latest g++ > > > qualify? I think this will have at least some impact on the public > > > reaction to your system. > > > > 2.7.2 is certainly nowhere near adequate. I don't think 2.8.0 is out > > yet, but early indications I have received make me fear it will also > > be inadequate, despite being a dramatic improvement over 2.7.2. > > > > We are aware this is a short term liability. Be we do believe it is a > > /short/ term liability, and in our group here at LLNL, we simply do > > not work with compilers that do not conform to a recent edition of the > > draft. It will be a great day indeed, when g++ is ANSI C++ > > compatible, and we look forward to that day, but we are not prepared > > to wait idly for that day. > > This one *is* meant to be a criticism. By requiring features that are not > in common distribution, you're cutting out a sizeable chunk of the user > base. LLNL may have pots of money to buy the latest optimizing C++ > compiler, but I am stuck either using the Sun CC compiler or g++. I don't > think that either is in full compliance with the ANSI draft standard, > although I'm not sure about the latest version of Sun's offering. > > I don't think that I am that atypical of academia, and I don't want to > wait a year or two for g++ to get up to par. If this is a mainstream > effort to get C++ support for python, it should be usable by mainstream > compilers. If this is really just a forum to perfect your idealized C++ > implementation, then there's probably enough demand for a separate "lowly > C++" SIG for everyone else. Reactions like this are inevitable. It is because I am sympathetic to such considerations that I put the fair-warning clause in the announcement. That way you don't waste time some day pulling down the patch set only to find you can't use it. At least now you know, without wasting any of your precious time. And time /is/ extremely precious in academia. It was only a year ago that I was still in academia. I was critically dependent upon g++. There is no conceivable way that I could've gotten through my degree program without a compiler at the level of g++. It was a life saver for me, literally. And your observation is right: at LLNL we can manage to buy compilers if we really want them. There are many ways to address the situation from your end. Not the least of which is to actively participate in g++ development. While I was using g++ daily, I personally: 1) Tested every release I could, including rotd's from cygnus. 2) Sent the g++ developers email at the rate of two to three messages per week, reporting bugs in as much detail as I could possibly manage. 3) Worked a contract to develop g++ support in a particular area. 4) Got on the g++ alpha users group for a particular platform I cared a lot about. 5) Learned enough RISC asm to help debug back end code generation bugs. If you care so much about g++, maybe you should do some of these things too. With respect to the hoity-toityness argument. The charter for this sig is very clear. It is for discussion by anyone, of anything relating to a C++ interface to Python. You do not need a new sig for discussing a lowball approach. You are welcome, and hereby so invited, to do it right here. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From fdrake at CNRI.Reston.Va.US Thu Feb 13 22:37:51 1997 From: fdrake at CNRI.Reston.Va.US (Fred L. Drake) Date: Thu, 13 Feb 1997 16:37:51 -0500 (EST) Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status In-Reply-To: <199702132121.QAA00933@anthem.CNRI.Reston.Va.US> from "Barry A. Warsaw" at Feb 13, 97 04:21:06 pm Message-ID: <199702132137.QAA02438@weyr.CNRI.Reston.Va.US> Even from the description of the work being done at LLNL, I'd guess that a much of effort to "wrap" the standard Python types with C++ classes can be done without using the (pricey) leading edge C++ compilers. My expectation would be that I could use the --with-cxx and wrappers with C++, that I could skip the test for new Thing() returning NULL if the runtime supports exceptions for failed new operations (does anyone know if g++ handles this these days, or if 2.8.0 will?), and the RTTI would only be needed for code generated by the "pythalyzer" tool that's been described. I think this is not be unreasonable; to support g++ for wrapped C++ libraries, the "pythalyzer" would need to generate different, perhaps more tedious, code, or impose a few restrictions on the classes that could be wrapped, but that is a livable situation. There's no reason that the pythalyzer can't have multiple "back-ends" of its own, so support could be provided for g++ wherever possible. (Yes, I expect that'd I'll only be using g++ for C++ code myself.) -Fred -- Fred L. Drake, Jr. fdrake at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive Reston, VA 20191-5434 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From phil at geog.ubc.ca Thu Feb 13 22:48:35 1997 From: phil at geog.ubc.ca (Phil Austin) Date: Thu, 13 Feb 1997 13:48:35 -0800 (PST) Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status In-Reply-To: References: <199702131737.MAA24664@python.org> Message-ID: <199702132148.NAA23916@coot.geog.ubc.ca> >>>>> "Johann" == Johann Hibschman writes: Johann> I don't think that I am that atypical of academia, and I Johann> don't want to wait a year or two for g++ to get up to par. Johann> If this is a mainstream effort to get C++ support for Johann> python, it should be usable by mainstream compilers. If Johann> this is really just a forum to perfect your idealized C++ Johann> implementation, then there's probably enough demand for a Johann> separate "lowly C++" SIG for everyone else. It's hard to imagine the traffic getting high enough to make that advisable, though. Since g++ will some day be ANSI standard, none of this effort is wasted, and those of us living on shoe strings can lurk and experiment with our 30 day evaluation copies of KCC ;-). Seeing excellent work being placed into the public domain by very capable DOE scientists noticibly lifts my spirits. (Speaking as an academic who hasn't paid US income taxes in about 10 years,) Thanks, Phil Phil Austin INTERNET: phil at geog.ubc.ca (604) 822-2175 FAX: (604) 822-6150 http://www.geog.ubc.ca/~phil Associate Professor Atmospheric Sciences Programme Geography #217 University of British Columbia 1984 W Mall Vancouver, BC V6T 1Z2 CANADA _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Fri Feb 14 04:19:38 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Thu, 13 Feb 1997 19:19:38 -0800 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <199702132021.PAA08910@zippy.boston.sgi.com> References: <199702122340.SAA19351@python.org> <199702132003.PAA00824@anthem.CNRI.Reston.Va.US> <199702132021.PAA08910@zippy.boston.sgi.com> Message-ID: <199702140322.WAA28174@python.org> Donald Beaudry writes: > "Barry A. Warsaw" wrote, > > Could you give me a quick primer on what the "traits" technique is? > > This is definitely something new to me. > > Check out , but I'm real > interested in hearing about how this is going to get applied to the > C++/Python issue. Here's a quick example. The following is the Foo::crash method I presented yesterday, augmented to pull out a couple of arguments from the args tuple. PyObject *Foo::crash( PyArgs *args ) { cout << "In Foo::crash\n" << flush; { int x, y; args->ParseTuple( x, y ); cout << "x=" << x << " y=" << y << endl; } throw "flaming oblivion"; Py_INCREF( Py_None ); return Py_None; } Notice several things: 1) No &'s !!! 2) No format string 3) No explicit error checking. Here are some possible ways one might call this thing from python. If you have the patience to study this closely, it will all make sense: mousey[34] ../python >>> import cxxtst >>> f = cxxtst.new_Foo() >>> f.crash( 2 ) In Foo::crash Traceback (innermost last): File "", line 1, in ? TypeError: function requires exactly 2 arguments; 1 given >>> f.crash( 2, 3 ) In Foo::crash x=2 y=3 Traceback (innermost last): File "", line 1, in ? RuntimeError: a C++ exception has occurred: flaming oblivion >>> f.crash( 2, 3, 4 ) In Foo::crash Traceback (innermost last): File "", line 1, in ? TypeError: function requires exactly 2 arguments; 3 given >>> f.crash( 2., "hello" ) In Foo::crash Traceback (innermost last): File "", line 1, in ? TypeError: illegal argument type for built-in operation >>> f.crash( 2., 3. ) In Foo::crash x=2 y=3 Traceback (innermost last): File "", line 1, in ? RuntimeError: a C++ exception has occurred: flaming oblivion >>> f.crash( 2.4, -6.6 ) In Foo::crash x=2 y=-6 Traceback (innermost last): File "", line 1, in ? RuntimeError: a C++ exception has occurred: flaming oblivion >>> f.crash( "xxx", 2 ) In Foo::crash Traceback (innermost last): File "", line 1, in ? TypeError: illegal argument type for built-in operation >>> So that seems like it does just what it should do, but is much less arduous to code than what you would do in C. Less arduous means I can do more python extension work without reaching for the bottle of Advil, and will be less likely to get anything wrong. Abolishing the format string is especially important. What if Foo was really template class Foo { ... } so that the "int x, y" in Foo::munge() above was really "T x, y;" ??? How would you handle that with format strings? This is just a rehashing of the same old argument which causes C++ to have an iostreams library instead of using stdio. I assume the import of this argument is clear to anyone with basic familiarity with using templates in C++. Now, how do we implement this? Here is the PyArgs class: //----------------------------------*-C++-*----------------------------------// // Copyright 1996 The Regents of the University of California. // All rights reserved. //---------------------------------------------------------------------------// #ifndef PyX_Args_hh #define PyX_Args_hh #include "Python.h" #include "PyX_xcpt.hh" #include template class pytraits { public: typedef PyObject *holder_t; static const char *fmt() { return "O"; } static void post_config( const holder_t& h, T& t ) { t = h; } }; template<> class pytraits { public: typedef int holder_t; static const char *fmt() { return "i"; } static void post_config( const holder_t& h, int& t ) { t = h; } }; template<> class pytraits { public: typedef float holder_t; static const char *fmt() { return "f"; } static void post_config( const holder_t& h, float& t ) { t = h; } }; template<> class pytraits { public: typedef double holder_t; static const char *fmt() { return "d"; } static void post_config( const holder_t& h, double& t ) { t = h; } }; //===========================================================================// // class PyArgs // The purpose of this class is to serve as the C++ replacement for the // "PyObject *args" which is the conventional second argument to compiled // extension functions. //===========================================================================// class PyArgs : public PyObject { public: int ParseTuple() { int result = PyArg_ParseTuple( this, "" ); if ( !result ) { throw PyException(); } return result; } template int ParseTuple( X1& x1 ) { pytraits::holder_t h1; std::string fmt = pytraits::fmt(); int result = PyArg_ParseTuple( this, const_cast(fmt.data()), &h1 ); if (!result) throw PyException(); pytraits::post_config( h1, x1 ); return result; } template int ParseTuple( X1& x1, X2& x2 ) { pytraits::holder_t h1; pytraits::holder_t h2; std::string fmt = pytraits::fmt(); fmt += pytraits::fmt(); int result = PyArg_ParseTuple( this, const_cast(fmt.data()), &h1, &h2 ); if (!result) throw PyException(); pytraits::post_config( h1, x1 ); pytraits::post_config( h2, x2 ); return result; } template int ParseTuple( X1& x1, X2& x2, X3& x3 ) { pytraits::holder_t h1; pytraits::holder_t h2; pytraits::holder_t h3; std::string fmt = pytraits::fmt(); fmt += pytraits::fmt(); fmt += pytraits::fmt(); int result = PyArg_ParseTuple( this, const_cast(fmt.data()), &h1, &h2, &h3 ); if (!result) throw PyException(); pytraits::post_config( h1, x1 ); pytraits::post_config( h2, x2 ); pytraits::post_config( h3, x3 ); return result; } }; #endif // PyX_Args_hh //---------------------------------------------------------------------------// // end of PyX_Args.hh //---------------------------------------------------------------------------// The plan is to have these member templates for each number of args from 0 up through some reasonable number, maybe 10 or 20. They are obviously going to get a little gross to write, but so what, we just put this grunge in the PyArgs class, and clients never have to think about it. Clients just say, "I need an int, a float, and a foobar out of that argument list", and so they declare an int, a float and a foobar, and ask the arg tuple to hand them over. A particularly nice thing about the "traits" technique, is that it is trainable. ParseTuple don't have no format code for your UniverseSimulator class? No problem, bury the conversion logic inside a specialization, pytraits. That is what the post_config business is for. You might need to pass an "O" for a given argument, since ParseTuple doesn't have format codes for things other than Python intrinsic types, and then you can use the post_config() method to do something meaningful about additional unpacking of the PyObject *. Check that it really is a UniverseSimulator, check that the universe hasn't died from heat death, or been sucked into a black hole, whatever, and then build up your C++ object from the PyObject *. You would do this by hand over and over without this, but with traits you just code up the conversion semantics one time, and presto--you've trained ParseTuple to know how to pick apart complex arg tuple strings with user defined extension types. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From pa at tekla.fi Fri Feb 14 10:02:30 1997 From: pa at tekla.fi (Harri Pasanen) Date: Fri, 14 Feb 1997 11:02:30 +0200 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <199702122340.SAA19351@python.org> References: <199702122340.SAA19351@python.org> Message-ID: <9702140902.AA28186@tahma.tekla.fi> Geoffrey Furnish writes: > > One of the first things which has to be faced is how to build a Python > which can support C++. On most systems, this means in particular, > that "main" must be compiled with the C++ compiler. Moreover, lacking > platform ABI's for C++, python must be built with a particular C++ > compiler--the one you will use for your extension work. You can't mix > and match .o's from C++ compilers the way you (often) can with .o's > from C compilers. Just a sidetracked field-report here: Under Linux, with g++, it is possible to build dynamic C++ extension modules that run fine under the vanilla autoconf built python. For instance, I have libwxmodule.so, which wraps large parts of the wxWindows C++ class library for wxPython. I would imagine this would work on other platforms with gcc/g++. Back to mainstream: I took a look at KAI C++ compiler page, and it seemed a Unix only product. Have you tried your stuff on NT platform? All in all, looks Cool. Harri PS. Have you tracked Jim Fulton's work on extension classes PSS. Are there any C++ compilers for Linux that are 'up to snuff'? If not, your stuff won't be usable on Beowulf clusters until g++ catches up... ;-) _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From bwarsaw at CNRI.Reston.Va.US Fri Feb 14 16:38:47 1997 From: bwarsaw at CNRI.Reston.Va.US (Barry A. Warsaw) Date: Fri, 14 Feb 1997 10:38:47 -0500 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status References: <199702122340.SAA19351@python.org> <199702132003.PAA00824@anthem.CNRI.Reston.Va.US> <199702132021.PAA08910@zippy.boston.sgi.com> <199702140322.WAA28174@python.org> Message-ID: <199702141538.KAA01493@anthem.CNRI.Reston.Va.US> Thanks for that example Geoffrey. Makes it very clear what the benefits are! _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Fri Feb 14 18:47:37 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Fri, 14 Feb 1997 09:47:37 -0800 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <9702140902.AA28186@tahma.tekla.fi> References: <199702122340.SAA19351@python.org> <9702140902.AA28186@tahma.tekla.fi> Message-ID: <199702141750.MAA02180@python.org> Harri Pasanen writes: > Geoffrey Furnish writes: > > > > One of the first things which has to be faced is how to build a Python > > which can support C++. On most systems, this means in particular, > > that "main" must be compiled with the C++ compiler. Moreover, lacking > > platform ABI's for C++, python must be built with a particular C++ > > compiler--the one you will use for your extension work. You can't mix > > and match .o's from C++ compilers the way you (often) can with .o's > > from C compilers. > > Just a sidetracked field-report here: > > Under Linux, with g++, it is possible to build dynamic C++ extension > modules that run fine under the vanilla autoconf built python. Yes, but this is because of the ELF file format, and is not really a general capability for systems XYZ. Also, you are probably not using templates, or are using one of the instantiation models which result in severe code bloat through replicate instantiation with static linkage. More sophisticated template closure schemes generally involve a speculative link phase with some sort of auto-construction technique for missing templates, finalized by a real link. For such models, you really do have to link with the C++ compiler. > For instance, I have libwxmodule.so, which wraps large parts of the > wxWindows C++ class library for wxPython. > > I would imagine this would work on other platforms with gcc/g++. Not necessary. You are depending on a feature of the ELF system, which the Linux developers (HJ and co) worked hard to perfect. > Back to mainstream: > > I took a look at KAI C++ compiler page, and it seemed a Unix only > product. Have you tried your stuff on NT platform? Send them email, you might be pleasantly surprised. > All in all, looks Cool. > > Harri > > PS. Have you tracked Jim Fulton's work on extension classes > > > PSS. Are there any C++ compilers for Linux that are 'up to snuff'? I am under the impression that KCC is supposed to be out for Linux sometime this spring, but that is rumor, I am not representing this statement as fact. > If not, your stuff won't be usable on Beowulf clusters until g++ > catches up... ;-) > I would like to respond to this again, even though there was quite a bit of discussion about it yesterday. The bottom line is that we at LLNL are building this C++ interface /because we need it/. We will build it whether other people like it or not. :-). However, if in the fortuitous event that it is useful to other people, we will be happy to make it available. We are willing to do what we can /reasonably/ do to facilitate use by others, but there are some sorts of actions which don't qualify under the "reasonable" rubric: 1) Expending lots of staff time to dumb down the interface so it will compile on standards-nonconformant compilers that we won't be using internally anyway. 2) Expending staff time to test these patches on every compiler on every OS on every hardware platform under the sun, etc. It is our belief that by writing it to the language standard, which will be ratified by ANSI and ISO imminently, that we will in so doing, create a package which is by definition, portable to every system of interest. If it happens that you have no way to get an ANSI C++ compiler on your computer, we're sorry. But we do not believe this is really a situation which will be of much /true/ concern, for very many people for very much longer. C++ is maturing rapidly. The last year in particular has seen dramatic improvements in the quality of compiler offerings by all the significant vendors. Many of them still have territory to make up, but we are sure they will. Barry asked about a survey of compiler vendors. We have effectively done this, although we don't have a written comparison chart. What we have are industry contacts in most of the major software houses, and pretty uniformly, vendors are promising support for conformance to the ANSI C++ standard, soon after ratification. Many vendors are currently, or will be shortly, shipping compilers with "draft standard conformance", which is a huge step up over the "barely better than Cfront" condition of the market only 1 year ago. I know it is tough when you're sitting in a degree program, strapped for cash, working for neandrathal advisors who claim that don't have money to buy you a single solitary commercial software product. There are tactics for coping with such situations, which ultimately have a lot to do with your personality and stamina for politics. Write a departmental memo to your academic dean, telling them your university is pathetically behind the times for not having modern language compilers. Go to faculty staff meetings and criticize in a public forum, the spending choices the faculty are making. Call up your vendor and ask about student specials, beta programs, and outright grants-to-the-poor. SGI has a really stupdendous program for academic support, called the "Varsity program". Get on it. If you school has other kinds of machines, call up your CS dept industry liason, and paint them into a corner if they don't have a site license agreement with every major workstation vendor. Bottom line, you don't have to take it sitting down. With specific reference to g++, don't tell me, tell Cygnus. They listen. G++ is under rapid development, and has led the industry in feature support for most of its product life. It happens to be a little behind the ball right now, since the vendors are finally starting to get serious, but this is no reason to give up hope for g++. It may be a reason to find an interim alternative. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From johann at physics.berkeley.edu Sat Feb 15 23:40:27 1997 From: johann at physics.berkeley.edu (Johann Hibschman) Date: Sat, 15 Feb 1997 14:40:27 -0800 (PST) Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status (fwd) Message-ID: My apologies to everyone if this is a duplicate, but I suspect that some system problems on my end ate this message the first time I tried to send it. It wasn't on the python.org archive when I checked, so here it goes again... Johann Hibschman johann at physics.berkeley.edu ---------- Forwarded message ---------- BTW, there's a real STL-related question buried at the end of this, for all you who have concluded that this debate is going nowhere... :-) On Thu, 13 Feb 1997, Geoffrey Furnish wrote: > Reactions like this are inevitable. It is because I am sympathetic to > such considerations that I put the fair-warning clause in the > announcement. That way you don't waste time some day pulling down the > patch set only to find you can't use it. At least now you know, > without wasting any of your precious time. And time /is/ extremely > precious in academia. Yes, and thank you for that warning. I understand your position in not wanting to waste *your* precious time on writing work-arounds for problems that your compiler doesn't have. However, I would like to see enough cooperation and awareness of the compiler issue to consider containing uses of things like member function templates and RTTI code to discrete portions of the code. Those discrete portions can then be worked around with elaborate hacks by the compiler-challenged. I don't know your code, so I don't know if this is already true, if this is nigh-impossible, or if this is possible with minor adjustments. > There are many ways to address the situation from your end. Not the > least of which is to actively participate in g++ development. Well, the Sun CC compiler I have access to here works better than g++, so I don't have too much experience in the innards of g++. But you're right: just whining isn't going to do anything, and I hope I didn't come across as saying that I expect the LLNL people to do everything for me. I appreciate the amount of work you've already done on this. I'm still climbing the C++ learning curve. I just looked into STL recently and was struck by the similarity of the feel of their standard classes to the standard Python data structures. If some of the template-intensive code can be replaced by STL calls, then end-users could simply use whatever STL implementation they may have at their site. That would confine some of the portability problems to a well-defined library which has probably already been ported. Personally, I was thinking about gradually implementing a STL-to-Python conversion, as I needed it in the code I'm writing for my own use. Does anyone think this is a good idea? If not, why not? Cheers, - Johann Hibschman P.S.: In any case, I have to be nice to you guys, since I will probably be attempting to borrow a beam simulation code from either the LBL beam people or the LLNL plasma folks once I finish deriving the distribution function. Heck, my advisor spends every other Friday or so down there, so I'll probably eventually get pulled along to say hi. --- Johann Hibschman johann at physics.berkeley.edu _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From skip at automatrix.com Mon Feb 17 17:18:57 1997 From: skip at automatrix.com (Skip Montanaro) Date: Mon, 17 Feb 1997 11:18:57 -0500 (EST) Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <199702140322.WAA28174@python.org> References: <199702122340.SAA19351@python.org> <199702132003.PAA00824@anthem.CNRI.Reston.Va.US> <199702132021.PAA08910@zippy.boston.sgi.com> <199702140322.WAA28174@python.org> Message-ID: <199702171618.LAA09227@dolphin.automatrix.com> >>>>> "Geoffrey" == Geoffrey Furnish writes: Geoffrey> Here's a quick example. The following is the Foo::crash method I Geoffrey> presented yesterday, augmented to pull out a couple of arguments from Geoffrey> the args tuple. Geoffrey> PyObject *Foo::crash( PyArgs *args ) Geoffrey> { Geoffrey> cout << "In Foo::crash\n" << flush; Geoffrey> { Geoffrey> int x, y; args-> ParseTuple( x, y ); Geoffrey> cout << "x=" << x << " y=" << y << endl; Geoffrey> } Geoffrey> throw "flaming oblivion"; Geoffrey> Py_INCREF( Py_None ); Geoffrey> return Py_None; Geoffrey> } ... Geoffrey> Now, how do we implement this? Here is the PyArgs class: ... How do you handle defaults and keyword arguments? -- Skip Montanaro | Python - it's not just for scripting anymore... skip at calendar.com | http://www.python.org/ (518)372-5583 | Musi-Cal ------> http://concerts.calendar.com/ _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From pa at tekla.fi Mon Feb 17 20:50:27 1997 From: pa at tekla.fi (Harri Pasanen) Date: Mon, 17 Feb 1997 21:50:27 +0200 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <266290262868014028@toto.iv> References: <199702122340.SAA19351@python.org> <9702140902.AA28186@tahma.tekla.fi> Message-ID: <9702171950.AA16122@tahma.tekla.fi> Geoffrey Furnish writes: > Harri Pasanen writes: > > > > PSS. Are there any C++ compilers for Linux that are 'up to snuff'? > > I am under the impression that KCC is supposed to be out for Linux > sometime this spring, but that is rumor, I am not representing this > statement as fact. > > > If not, your stuff won't be usable on Beowulf clusters until g++ > > catches up... ;-) > > > > I would like to respond to this again, even though there was quite a > bit of discussion about it yesterday. > [ rest of of the reasoning deleted ] I did not mean to criticize your approach, all of your reasoning is quite solid. In fact, I'd take the same approach you took being in the same situation. I was also just reading about the Beowulf project, and wanted to give a pointer to that stuff as a sideline. (and got Livermore and LANL (Los Alamos) mixed up.) Do you have an estimate of when you'll be releasing your stuff to wider audience? Harri _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Tue Feb 18 14:13:46 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Tue, 18 Feb 1997 05:13:46 -0800 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <199702171618.LAA09227@dolphin.automatrix.com> References: <199702122340.SAA19351@python.org> <199702132003.PAA00824@anthem.CNRI.Reston.Va.US> <199702132021.PAA08910@zippy.boston.sgi.com> <199702140322.WAA28174@python.org> <199702171618.LAA09227@dolphin.automatrix.com> Message-ID: <199702181316.IAA28062@python.org> Skip Montanaro writes: > How do you handle defaults and keyword arguments? Haven't yet, suggestions welcome. I was figuring there would have to be two method invocation trampolines, one for keyword arguments, and one for no keyword arguments, just as is done with the built in global function invocation mechanism. But that's not a very complete answer to your question, I realize. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Tue Feb 18 14:17:31 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Tue, 18 Feb 1997 05:17:31 -0800 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <9702171950.AA16122@tahma.tekla.fi> References: <199702122340.SAA19351@python.org> <9702140902.AA28186@tahma.tekla.fi> <266290262868014028@toto.iv> <9702171950.AA16122@tahma.tekla.fi> Message-ID: <199702181320.IAA28103@python.org> Harri Pasanen writes: > I was also just reading about the Beowulf project, and wanted to give > a pointer to that stuff as a sideline. (and got Livermore > and LANL (Los Alamos) mixed up.) No problem, it was a great read. Everyone should have one of those things in their group... Besides, there is at least one person on this list who has access to that machine. > Do you have an estimate of when you'll be releasing your stuff to > wider audience? I'll take that as a formal request, and we'll see about doing a code drop. Will get back to the list when we have a better idea. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Tue Feb 18 15:06:58 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Tue, 18 Feb 1997 06:06:58 -0800 Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status (fwd) In-Reply-To: References: Message-ID: <199702181410.JAA28328@python.org> Johann Hibschman writes: > However, I would like to see enough cooperation and awareness of the > compiler issue to consider containing uses of things like member function > templates and RTTI code to discrete portions of the code. Those discrete > portions can then be worked around with elaborate hacks by the > compiler-challenged. The direct invocation of C++ member functions should work with an compiler at the level of Cfront 2.1 or better. I know of only one computer in the world where the situation is still this bad (name withheld to protect the guilty). However, the argument parsing in our plan uses member templates. I could imagine a non-member template plan working something like this: template void PyArgs_ParseTuple( PyArgs *args, X1& x1, X2& x2 ); You would then do something like: PyArgs_ParseTuple( args, x1, x2 ); instead of what we will do: args->ParseTuple( x1, x2 ); This would rely on only regular templates, not member templates, and would help people get through the pinch for a while. The traits technique in principle is not beyond the capability of Cfront, but the new-style specialization syntax is not known by Cfront. Not sure what to say about that. I'm not sure if there is a way to state template specializations which is admissible under both the new ANSI standard and under the old ad-hoc conventions. A language lawyer I am not. > I don't know your code, so I don't know if this is already true, if this > is nigh-impossible, or if this is possible with minor adjustments. It's hard for me to say. I think it will be necessary for people who want workalikes for nonconformant compilers to spend a little time working up the files to provide the alternate mechanisms. When someone tries this, they can tell the list whether it was merely annoying, or practically death-defying. > I'm still climbing the C++ learning curve. I just looked into STL > recently and was struck by the similarity of the feel of their > standard classes to the standard Python data structures. If some > of the template-intensive code can be replaced by STL calls, then > end-users could simply use whatever STL implementation they may > have at their site. That would confine some of the portability > problems to a well-defined library which has probably already been > ported. One way of reading this would be to think of trying to build an STL adaptor to a PyList, or some such. I have thought of that too, but have not acted on it. We are still learning the STL techniques here, and so could still use suggestions. Since we plan to make classes to wrap the Python C API anyway, it seems like the most obvious plan is to simply build these classes with iterators which obey stl-like semantics. But until there are some sample implementations of the various competing ideas, it will be hard to pass judgement on the quality of the various approaches. > Personally, I was thinking about gradually implementing a STL-to-Python > conversion, as I needed it in the code I'm writing for my own use. Does > anyone think this is a good idea? If not, why not? Can you define this more precisely? I can't quite tell what you mean. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Tue Feb 18 15:21:47 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Tue, 18 Feb 1997 06:21:47 -0800 Subject: [PYTHON C++-SIG] LLNL Python/C++ integration: current status In-Reply-To: <199702132003.PAA00824@anthem.CNRI.Reston.Va.US> References: <199702122340.SAA19351@python.org> <199702132003.PAA00824@anthem.CNRI.Reston.Va.US> Message-ID: <199702181424.JAA28409@python.org> Catching up on some email traffic that hit me in a cruch last week. Barry A. Warsaw writes: > > First an aside: I have not tracked C++ developments very closely since > coming here to CNRI about 2 1/2 years ago, so I'm only dimly aware of > details of the new features like RTTI, exceptions, and namespaces. I > used to have one of the earliest editions of the ARM. Would you > recommend getting a new edition, or is there a better reference manual > I can get to bone up on these features? I'm not a language lawyer, so the actual ARM hits me pretty dry. I have found, however, that Stroustrup, "Design and Evolution of C++" is extremely readable, and provides good discussion of the rationale for the fatures which have shown up in the language, as well as detailed discussion of many of the proposals which were rejected, and /why/ they were rejected. Very little will give you the breadth of coverage of C++ that D&E provides. Other stuff: We recently bought a bunch of copies of Scott Meyer's two books, "Effective C++", and "More Effective C++", and passed them around to all members of our team. I have found both of these to be really excellent. Another excellent choice is the "C++ FAQ" by Marshal Cline. He hits /lots/ of the pragmatics which go understressed in much of the C++ literature. > > 3) Introducing C++ classes for the various major Python abstractions, > > such as dictionaries, lists, tuples, modules, etc. > > GF> Python really look this way to the C++ programmer. Planned > GF> classes at this time include: > > GF> PyDict > GF> PyList > GF> PyTuple > > GF> and possible also PyArgs, > > I'd only suggest adding the word `Class' on the end here, (or maybe > `Wrapper') e.g. PyDictClass. Although `PyDict' may never conflict > with other names in the C API, there are symbols like `PyDict_Type'. > It seems like it could cause confusion, for example, what if you > passed a PyDict object to PyDict_Type? That would seem to me a > natural thing to want to do given the naming. I haven't looked at that exact function yet, but I am imagining that it will be shadowed by a a method in PyDict. so you wouldn't actually "pass a PyDict to PyDict_Type". Instead you would "invoke the method Type() on a PyDict". PyDict *ed; d->Type(); Also, all the C++ stuff is going to live in a namespace, so actual name clashes should not be a problem. I am a little bit inclined to jus go ahead and drop the Py prefixes, since they will all live in a namespace which will probably be named Py. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From johann at physics.berkeley.edu Wed Feb 19 08:16:07 1997 From: johann at physics.berkeley.edu (Johann Hibschman) Date: Tue, 18 Feb 1997 23:16:07 -0800 (PST) Subject: [PYTHON C++-SIG] Python/STL thoughts In-Reply-To: <199702181410.JAA28328@python.org> Message-ID: On Tue, 18 Feb 1997, Geoffrey Furnish wrote: > Johann Hibschman writes: > > Personally, I was thinking about gradually implementing a STL-to-Python > > conversion, as I needed it in the code I'm writing for my own use. Does > > anyone think this is a good idea? If not, why not? > > Can you define this more precisely? I can't quite tell what you mean. Well, as I see it, there are three ways to think of integrating Python and C++ data structures: 1. For primarily C++ programs, a program should be able to take STL container objects and copy them into Python lists or dictionaries, allow the user to manipulate these objects, and then copy the Python representation into a native C++ representation. 2. For Python extensions that happen to be written in C++, the program should be able to create an STL-like object which manipulates the actual Python list/dictionary through calls to PyList_Insert, PyList_GetItem, etc., without actually copying the container over into a C++ native form. i.e. the STL list contains type PyObject *, which the program must then explicitly handle. 3. Create a C++ extension which properly defines the Python sequence or dictionary methods, so the native C++ representation internal to the code is manipulated by Python-level commands. I was thinking of (1), namely a way to easily make data structures in a normal C++ program visible to Python without requiring major changes to the original C++ program. This would be for interactive debugging and data monitoring use. (Eventually I'd want to do this to NumPy arrays, but if I can get it into a list, I can get it into an array.) Category (2) is the foundation for most of the work in (1). I envision being able to do something like: pythonlist = ; copy( pythonlist.begin(), pythonlist.end(), cpplist.begin() ); to copy over data items from a python object to a c++ object. Of course, the problem is now how to tell the python list what kind of object it supposed to contain, since the c++ lists want homogeneous objects, while a python list could contain just about anything. Perhaps an STL-style iterator adaptor would be appropriate, as in copy( pythonlist.begin(), pythonlist.end(), PyAdaptor(cpplist.begin()) ); where the "PyAdaptor" would contain the logic to cast the PyObject pointers in the Python list to doubles, and raise an exception if any of the objects in the list was not a double. Category (3) is mostly for full-featured language extensions, although a Python wrapper for the STL container classes might be useful. For example, maybe a linked-list representation of a list might be more efficient for some task than the standard Python array implementation, since it would allow efficient insertions and deletions. A Python wrapper which uses STL for its functionality could do that fairly easily. As you can see, I have a number of ideas about how this could work, but few specifics. I'd appreciate anyone's comments about what they think is the most common role of transferring data in Python extensions, about the feasibility of using STL-based containers, and about any relevant ideas whatsoever. Cheers, - Johann --- Johann Hibschman johann at physics.berkeley.edu _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From fdrake at CNRI.Reston.Va.US Wed Feb 19 17:09:53 1997 From: fdrake at CNRI.Reston.Va.US (Fred L. Drake) Date: Wed, 19 Feb 1997 11:09:53 -0500 (EST) Subject: [PYTHON C++-SIG] Another article that mentions the "traits" idiom Message-ID: <199702191609.LAA18487@weyr.CNRI.Reston.Va.US> For those who just learned about the "traits" idiom, here's a pointer to another article that talks a little about it: http://www.sigs.com/publications/docs/cppr/9612/cppr9612.kreft.html -Fred -- Fred L. Drake, Jr. fdrake at cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive Reston, VA 20191-5434 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From johann at physics.berkeley.edu Wed Feb 19 22:38:23 1997 From: johann at physics.berkeley.edu (Johann Hibschman) Date: Wed, 19 Feb 1997 13:38:23 -0800 (PST) Subject: [PYTHON C++-SIG] Re: LLNL Python/C++ integration: current status (fwd) In-Reply-To: <199702181410.JAA28328@python.org> Message-ID: On Tue, 18 Feb 1997, Geoffrey Furnish wrote: > Johann Hibschman writes: > > However, I would like to see enough cooperation and awareness of the > > compiler issue to consider containing uses of things like member function > > templates and RTTI code to discrete portions of the code. Those discrete > > portions can then be worked around with elaborate hacks by the > > compiler-challenged. > > The direct invocation of C++ member functions should work with an > compiler at the level of Cfront 2.1 or better. I know of only one > computer in the world where the situation is still this bad (name > withheld to protect the guilty). Right. There shouldn't be any problem with that. > However, the argument parsing in our plan uses member templates. I > could imagine a non-member template plan working something like this: > > template > void PyArgs_ParseTuple( PyArgs *args, X1& x1, X2& x2 ); > > You would then do something like: > > PyArgs_ParseTuple( args, x1, x2 ); > > instead of what we will do: > > args->ParseTuple( x1, x2 ); > > This would rely on only regular templates, not member templates, and > would help people get through the pinch for a while. I'd prefer to keep the programmer API the same, and find some good way to modify the PyArgs class to work without changing its interface. Perhaps an additional class could mediate the call. On a similar note, is the static cast in PyArgs *arglist = static_cast( args ); really necessary? Couldn't the constructor PyArgs class simply take a PyObject pointer argument instead, like PyArgs *arglist = new PyArgs ( args ); or is there some reason this won't work? - Johann --- Johann Hibschman johann at physics.berkeley.edu _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From johann at physics.berkeley.edu Fri Feb 21 22:57:47 1997 From: johann at physics.berkeley.edu (Johann Hibschman) Date: Fri, 21 Feb 1997 13:57:47 -0800 (PST) Subject: [PYTHON C++-SIG] Typecasts to/from PyObject? Message-ID: Hi all, I have a style question. If I define a class around the usual PyObject pointer, like class CPyObject { PyObject *obj; public: CPyObject( PyObject *o ) : obj(o) {} } would it be a good idea to also define C++->CPyObject data conversions, like adding, CPyObject( double x ) { obj = PyBuildValue("d", d); } or CPyObject->C++ conversions, like operator double() { double x; PyArg_Parse( obj, "d", &x ); // add appropriate error handling... return x; } or would it just provide a source of confusing bugs? I'm looking at this in the context of creating a PyDict class. If I define the type conversions, I could write: PyList plist; plist["spam"] = 3.0; x = y + plist["spam"]; while otherwise I would have to write something like plist[ toPyObject("spam") ] = toPyObject(3.0); x= y + PyObjectToDouble( plist[ toPyObject("spam") ] ); Conversions are dangerous, in that they might do unexpected things, but they would make the syntax a lot nicer. As an example of the bad effects, if a program had a reference to a floating-point python object, like CPyObject pi( PyBuildValue("d", 3.14159 ) ); then this code would give an error of some kind (I believe): double x, y=3.14; int i, j; x = 2.0 + pi; // fine, converts CPyObject to double x = 2 + pi; // Would this try to convert CPyObject to int or double? i = 2 + pi; // Would the attempted CPyObject to int conversion work? i = 2 + y; // This is fine, but loses information... Opinions? - Johann P.S. Yes, in a real version, these conversions would probably be implemented through traits-style templates, but I haven't quite thought my way all the way through that yet. Johann Hibschman johann at physics.berkeley.edu _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From furnish at laura.llnl.gov Mon Feb 24 17:45:37 1997 From: furnish at laura.llnl.gov (Geoffrey Furnish) Date: Mon, 24 Feb 1997 08:45:37 -0800 Subject: [PYTHON C++-SIG] Typecasts to/from PyObject? In-Reply-To: References: Message-ID: <199702241648.LAA09245@python.org> Johann Hibschman writes: > Hi all, > > I have a style question. If I define a class around the usual PyObject > pointer, like > > class CPyObject { > PyObject *obj; > public: > CPyObject( PyObject *o ) : obj(o) {} > } > > would it be a good idea to also define C++->CPyObject data conversions, > like adding, > > CPyObject( double x ) { obj = PyBuildValue("d", d); } > > or CPyObject->C++ conversions, like > > operator double() { > double x; > PyArg_Parse( obj, "d", &x ); // add appropriate error handling... > return x; > } > > or would it just provide a source of confusing bugs? I think it is a good idea, so we intend to do things like this. > I'm looking at this in the context of creating a PyDict class. If I > define the type conversions, I could write: > > PyList plist; > plist["spam"] = 3.0; > x = y + plist["spam"]; That is nice, if it works. However, I am not sure how you avoid having multiple available conversions, and what effect that has on the matching rules. > while otherwise I would have to write something like > > plist[ toPyObject("spam") ] = toPyObject(3.0); > x= y + PyObjectToDouble( plist[ toPyObject("spam") ] ); That is something we would like to avoid. > Conversions are dangerous, in that they might do unexpected things, but > they would make the syntax a lot nicer. One way to approach this is to mark the potentially ambiguous conversions explicit, so that the compiler won't use them implicitly. Then the user would have to use the conversions directly, and this would moreover, provide a place for an assertion. > As an example of the bad effects, if a program had a reference to a > floating-point python object, like > > CPyObject pi( PyBuildValue("d", 3.14159 ) ); > > then this code would give an error of some kind (I believe): > > double x, y=3.14; > int i, j; > x = 2.0 + pi; // fine, converts CPyObject to double > x = 2 + pi; // Would this try to convert CPyObject to int or double? > i = 2 + pi; // Would the attempted CPyObject to int conversion work? > i = 2 + y; // This is fine, but loses information... > > Opinions? class CPyObject { // internal state public: explicit operator double() { // Check that it really is a double. return ; } }; I think this would result in code like: x = 2.0 + double(pi); x = 2 + int(pi); both of which could bbe made to work robustly and are much less ugly than the equivalent C code. > - Johann > > P.S. Yes, in a real version, these conversions would probably be > implemented through traits-style templates, but I haven't quite thought > my way all the way through that yet. Please elaborate. -- Geoffrey Furnish email: furnish at llnl.gov LLNL X/ICF phone: 510-424-4227 fax: 510-423-0925 _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________ From johann at physics.berkeley.edu Thu Feb 27 03:05:55 1997 From: johann at physics.berkeley.edu (Johann Hibschman) Date: Wed, 26 Feb 1997 18:05:55 -0800 (PST) Subject: [PYTHON C++-SIG] Typecasts to/from PyObject? In-Reply-To: <199702241648.LAA09245@python.org> Message-ID: Geoffrey Furnish writes: > Johann Hibschman writes: > > I'm looking at this in the context of creating a PyDict class. If I > > define the type conversions, I could write: > > > > PyList plist; > > plist["spam"] = 3.0; > > x = y + plist["spam"]; > > That is nice, if it works. However, I am not sure how you avoid > having multiple available conversions, and what effect that has on the > matching rules. As you mention, having both a double conversion and an int conversion in the same class would confuse the compiler & require an explicit cast. I was thinking more along the lines of having several types. First, a generic PyObj type which only supports implicit conversion to and from a PyObject *, and which tracks the reference count of the object pointed to automatically. For compilers that can do member function templates, it could also do explicit casts to any other type with defined python traits. All in all, it's pretty straightforward. It would be initialized from a PyObject *. Then, I was thinking of having a templated class PyType, either subclassed off of the PyObj class, or subclassed off of the same abstract class as PyObj, which would support conversion to/from a PyObject *, as well as conversion to/from type T. The conversion code could then be stored in the pytraits class. (Question: would it be better to convert to/from T*, rather than T? These could then by PyPtr classes, since in many ways they are similar to ideas of smart pointers.) The PyObj class would be used when no assumptions are being made about the type of the object, while the PyType's would make the assumption that the python object was in fact of type T. For lists, a templated adaptor class could be constructed fairly easily which would provide a default return type for the list, so you could write: intlist = PyAdapt(pylist); x = 2*intlist[2]/intlist[3]; Basically, the class would define a non-explicit operator T() method, to allow automatic conversion. Since the lists would have to implement a proxy class to handle the difference between using operator[] in an lvalue or rvalue context, each adaptor would probably have to implement its own (templated) proxy class. Having babbled that much, a sketch of an implementation would look like: // --- types --- class PyAbstractType { public: PyAbstractType( PyObject *x ) { p = x; } ~PyAbstractType() = 0; operator (PyObject *)() { return p; } // reference counting machinery private: PyObject *p; } // pure virtual destructor definition. decrement reference count? PyAbstractType::~PyAbstractType() {} // allow default cast to PyObject, or explicit cast to anything... class PyObj : public PyAbstractType { public: PyObj( PyObject *x ) : PyAbstractType(x) {} template // my own compiler can't do this... explicit operator T() { return pytraits::fromPyObj(p); } } template class PyType: public PyAbstractType { public: PyType( PyObject *x ) : PyAbstractType(x) {} PyType( const T &x ) : PyAbstractType( pytraits::toPyObj(x) ) {} operator T() const { return pytraits::fromPyObj(p); } } // -- lists -- // uses casting to/from PyObject * ability of PyList, PyType // needs error checking, must check syntax for SetItem/GetItem // must also fiddle a bit for const-correctness template class PyListProxy { public: int index; PyList &lst; PyListProxy( PyList &l, int i ) : lst(l), index(i) {} // use as lvalue PyType operator= ( const T &ob ) { PyType temp(ob); PyList_SetItem( lst, index, temp ); return temp; } // use as lvalue 2 (probably has bugs) PyAbstractType &operator= ( const PyAbstractType &ob ) { PyList_SetItem( lst, index, ob ); return ob; } // use as rvalue (PyType can convert to T) operator PyType () { return PyType( PyList_GetItem( lst, index ) ); } } // non-template version, works on PyObj's class PyListProxy { public: int index; PyList &lst; PyListProxy( PyList &l, int i ) : lst(l), index(i) {} // use as lvalue PyObj operator= ( const PyObj &ob ) { PyList_SetItem( lst, index, ob ); return ob; } // use as rvalue operator PyObj () { return PyObj( PyList_GetItem( lst, index ) ); } } // actual list class PyList : public PyAbstractType { public: PyList( PyObject *x ) : PyAbstractType(x) {} // maybe a templated constructor from STL list? PyListProxy operator[](int index) { return PyListProxy( *this, index ); } } // template version = same as above, but uses PyListProxy's Okay, that's enough for now. That was a longer sketch than I thought it would be, and it is all just off the top of my head, so it probably has quite a few mistakes, but it gives an idea of what I'm thinking of. I don't have the time to code it for real, test it out, etc. As soon as I get out of my current analytic phase and back into numeric calculations, I'll have the incentive to hack it out for real, but for now I have other things on my mind. Feedback? - Johann -- Johann Hibschman | Grad student in Physics, lurking in Astronomy, johann at physics.berkeley.edu | Probing Pulsar Pair Plasma Production Processes _______________ C++-SIG - SIG for Development of a C++ Binding to Python send messages to: c++-sig at python.org administrivia to: c++-sig-request at python.org _______________