SWIG and Callbacks

[Jim]

David Abrahams wrote:
> "Jim" <jbublitzNO at SPAMnwinternet.com> wrote in message
> news:3D0D3223.5070500 at SPAMnwinternet.com...

>>The biggest plus for sip is that the 'IDL' is the h file syntax
>>with a few subtractions (variable names, forward declarations,
>>inline code, but inline methods stay), so that most of the
>>description file creation is easy to automate and still not
>>that hard to do manually for a small number of files. sip
>>supports most C++ h file syntax.

> Hmm, I wonder about that. The C++ type system is pretty sophisticated (to
> the point of being able to represent entire compile-time programs!) and all
> of the parser-based wrapping systems I've seen eventually fail to handle
> some constructs I use, especially if I'm wrapping template-based code.
> That's part of the reason I chose to use the compiler itself in
> Boost.Python. The compiler (in theory) has no problem with C++ syntax ;-)

> On the other hand, I realize there's a large category of simple C++
> interfaces which can be parsed will less-than-complete machinery.

Template-based code does require special treatment with sip.
In most cases where I've wrapped template-based types or
classes, I manually write C++ code for conversion C++ <-> Py
(sip has some functions that make that easier) and implement
it as a sip 'mapped type', which lets me refer to the same
conversion code from any return value or arg. I generally
assign a new name (eg QList<int> -> IntList) but wouldn't
have to (sip will parse QList<int> and look for a mapped
type definition corresponding).

sip won't handle all possible C++ syntax, and it sometimes
takes some creativity to shoehorn something in, but something
well over 80% of h file code translates directly (after
removing variable names), and only 1 or 2% really takes
major effort.

The real problem wouldn't seem to be C++ types as such, but
finding a useful (to the Python programmer) representation
via PyObject, and still maintaining the range of abilities
built into the C++ code (esp things like subclassing).

>>The other features that are helpful on large projects are the
>>built-in support for versioning

> How so? Can you say more about this?

sip uses directives: %If (A - B) ... %End to wrap a block
of code for versioning over a range. It could also be
%If (A - ) (>= A) or %If ( - B) (< B). Acceptable values
for A and B are defined in a %Timeline declaration; actual
values for A and B are passed to sip via the command line,
usually  derived from a version #define in the C++ source
code via configure (at least on Linux). Evaluation is by
lexical ordering of the A and B values. A and B are
strings like 'KDE_3_0_0'.

Not very different from #ifdefs, but maybe a little
easier.

>>and embedded docs (both also fairly
>>easy to automate),

> Do you mean docstring generation?

No - similar, but sip provides %ExportDoc ... %End directives
inside which you could place just about anything you want I
suppose. For PyQt/PyKDE the directives wrap SGML marked-up text.
You can construct an input file which along with the proper sip
command line switch, will collect and concatenate all of the
doc blocks distributed over the description files. The
concatenated file can be processed (in the case of SGML) to
provide HTML docs, or whatever you require.

>>as well as support for 'platforms' and
>>'features'. So for example PyQt (not my project) can support Qt
>>from 1.4.2 through 3.0.4 with different mixes of features (with
>>SQL or not, QTable or not) across Windows, Linux/Unix/BSD, and
>>(soon) Mac, with a single set of source files, and still be
>>manageable by one person.

> Boost.Python works on a similar range of platforms.
> I'm not sure if this configuration stuff is an appropriate job for a
> wrapping system to provide specific support for; what's the argument for
> integrating it with the wrapping system?

PyKDE consists of about 400 description files spread over 10-12
libraries depending on version and not necessarily with the same
description file <-> library correspondence in each version. PyQt
is slightly smaller, but supports more platforms. The alternative
to embedding version/platform/feature info in the description files
is to maintain separate sets of files. In the case of PyKDE (depends
on PyQt) for KDE < 3.0.0, it might build with 2.3.0 <= Qt <= 2.3.2 
depending on what the user has. If the user compiled Qt himself, he 
might have a different feature set than the official distributed Qt
for his platform. The number of permutations is fairly large and that
would be multiplied by the number of files.

On the other hand, changes or bug fixes are usually the same across
versions/platforms/features. So in addition to only supporting one
fileset instead of many, it's only necessary to repair/edit one
fileset instead of many. Similarly, it isn't possible to test every
possible platform - the potential for simple typos crippling a
particular version alone is pretty large, and most code is common.

Again, #ifdefs probably provide similar functionality - this is
just a little more formalized/standardized and supported by
the tools (so error checking is possible, for instance).

>>sip is flexible enough that you can handwrite code to "Pythonize"
>>C++ code (return a Python list or dict, or for example,
>>'void rgb (int&, int&, int&)' returns (int, int, int) and takes
>>no arguments in PyKDE), or create type substitutions/conversions
>>('mapped types') once and re-use them across a project. You can
>>do C++ with anything of course, but sip has it well-integrated
>>(manageable) and has support code to make it easier (arg unpacking,
>>object conversions, etc). IIRC, Boost might not need handwritten
>>C++ code quite as much.

> Not sure about this; we'd need to compare specific functionality.

My guess is that Boost does at least some of this a little more
naturally. I'm not sure the extent that Boost can modify things
like return values or args without resorting to C++, however. sip
has to resort to C++, but has helper functions, so a typical
piece of handwritten code rarely exceeds 10 - 15 lines, including
declarations.

>>I suppose those are a series of small advantages, but the end
>>result seems to me that you write less code

> Are you counting the IDL file? That's code, IMO.

Yes, but I don't write it - I just edit it mostly to correct
mistakes my description file generator creates and to handwrite
code where the generator has flagged that. Crude but effective (TM)
Even without automation, the process is easiest if you start by
saving the h file with a .sip extension and then (mostly) delete
the things that sip doesn't want/need, so it's still mostly
editing (sometimes A LOT of editing).

>>Boost (IIRC) also has a big advantage in being able to bind much
>>more selectively.

> Can you explain what you mean by that?

I should probably go back and review Boost. I had thought that
with Boost it was possible to wrap as much or as little of a
C++ library as you want to. That's more of a problem with sip.

>>sip is easiest when binding everything in a lib
>>and probably takes a little more effort to be selective - you
>>can do it, but it isn't as 'mindless' (ie automatable) as most
>>sip file creation is. And of course Boost is actually
>>documented - no small thing.

> Heh, yeah. Unfortunately, documentation is often harder than coding. Even
> the current official release of Boost.Python is not very well-documented.
> We're working hard to correct that for v2, which is in pre-release.

Well, you have docs ... I found the web site to be pretty good at
explaining how to use Boost (and perhaps haven't remembered it all
correctly).

>>How would you see Boost performing on a very large project -
>>say 10 libs, 20 classes/lib, 10 exposed methods/class (2000
>>methods overall)?

> See http://cctbx.sourceforge.net/
> Also
> http://mambo.peabody.jhu.edu/~karlmac/publications/gamera_python_2001/gamera
> .html

> Of course, these don't prove anything, since wrapping is a nontrivial job
> and people quickly get invested in whatever is working for them. You *can*
> wrap a large project with Boost.Python, but whether it's easier than with
> sip...?

Never having used Boost, I couldn't say either. Bindings do
seem to require a particular mindset. There is a learning curve
certainly, and as mentioned, familiarity with a particular
method introduces a lot of expectations.

>>I haven't looked at Boost in a while -
>>could you incorporate conditional compilation (eg versions,
>>features)

> It's C++; you can use #ifdef in just the same way that the rest of the C++
> code surely does.

sip's only advantage then is that it's able to do some error
checking on versioning/feature conditionals because the
syntax is built into sip.

>>and embed docs easily?

> Docstring support is on the todo list for v2. It will be a matter of passing
> a string literal to the def() function which wraps a function pointer:

> class_<Foo>()
>    .def_init(args<int>(), "Creates a Foo with the specified number of Bars")
>    .def("get_name", &Foo::get_name, "get the Foo's name")
>    .def("set_name", &Foo::set_name, "set the Foo's name")
>    ;

That looks cool - it's a little more cumbersome with
sip.

>>Could you automate code generation to some extent?

> Not sure what you have in mind here. In a sense, the metaprogramming
> techniques used by Boost.Python already *do* code generation.

Yeah - 'code generation' is ambiguous. sip itself is a code generator
(it produces C++/Python files which are the actual bindings or Python
lib). What I meant in this case is generating the description files
directly from the h files. I currently take a set of h files, run them
through a handwritten parser and generate *.sip files which sip then
uses to generate the actual C++ code for the bindings. I had the
impression SWIG was supposed to do most of that in a single step. Just
wondering if I could implement something like that with Boost, or
need to code a line manually for each method/class/whatever.

>>The automation for generating
>>PyKDE is pretty crude at the moment, and in all fairness
>>isn't generally available, so this example would be a lot
>>of work using sip as it comes out of the box. Is Boost
>>really intended for projects of this size (there probably
>>aren't many), or do you have different goals for it?

> Yes, Boost.Python is intended for large projects. One of the largest
> projects I've ever heard is using it at Lawrence Livermore Labs, for
> example.

It's pretty amazing to find out who's using Python and things
like Boost, pyGTK. PyQt, PyKDE, etc.

I left your web site with the impression that I (personally) would
find sip easier for large projects, but Boost would have advantages
on smaller projects (unfortunately I remember the impression better
than the reasons for it). That's especially interesting to me at
the moment, because I have ideas for a lot of smaller projects. I expect
I'll stick with sip, but will have to take another look at Boost, if
only to steal good ideas :)

When I get a chance, I'll check out the links above.

Jim