[C++-sig] New Major-Release Boost.Python Development

[Dave Abrahams]

Trying to catch up here, so responding to everything all at once.

on Thu Aug 25 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:

Just how tall are you, Jimbo?

> I'd like to start work on a new major release of Boost.Python.  

That certainly is welcome news.

> While the library is currently well-maintained in terms of bugfixes, I
> get the sense that neither the original developers nor the current
> maintainer have the time or inclination to work on new features.  

Well, speaking for myself, mostly time.  I'd be inclined to do a rewrite
along the lines of the langbinding ideas if I had time.

Another thing I think should be examined and refreshed is the
documentation style.

> I'd also like to propose some changes that are slightly
> backwards-incompatible, as well as some that mess with the internals
> to an extent that I'd feel better about doing it outside Boost itself,
> to make it easier for adventurous users to play with the new version
> without affecting people who depend on having an extremely stable
> library in Boost.

There's no need to do this "outside of Boost."  A branch in the Boost
repository is a perfect place for exploratory development that will
eventually be released as part of Boost.

> To that end, I'm inclined to copy the library to somewhere else
> (possibly the boost sandbox, but more likely a separate site), work on
> it, produce some minor releases, and re-submit it to Boost for
> review. 

If you want to go through another review process, that's up to you.
Getting review feedback definitely has its advantages.  Please, though,
use the sandbox or some other area of the Boost svn repository at least
until we get my hoped-for Git transition completed.

> Perhaps the external site would continue on as the home of more
> fine-grained releases, or maybe we would fully reintegrate with Boost
> at that point (especially if Boost addresses some of its own project
> management and release control issues by that point, which I know is
> being discussed but to my knowledge doesn't really have a timeline
> yet).
>
> I am willing to take the lead on this project; 

Someone willing to take the lead is the most important part.

> I have a number of features that exist as extensions in the boost
> sandbox already that would work better if they could be more fully
> integrated into the Boost.Python core, 

Good.  Although, if I were you I would also carefully re-examine the
core to see if it has the best design.

> and I think I have the necessary understanding of the full code base
> to coordinate things.

Awesome.

> I'd like to save a full discussion of what features a new version
> would include for another thread, but I am hoping other people on the
> list might volunteer some time to work on aspects they have coded up
> elsewhere - I know many such extensions exist.
>
> So I have a few questions for anyone who's paying attention:
>
> - For the original Boost.Python developers and current maintainers,
> and other people familiar with developing Boost libraries: do you have
> any preference on how to approach this?  I don't want to step on any
> toes, especially toes attached to people who are responsible for the
> excellent library we already have.

See above.  No toe-stepping concerns on my end.  I have a preference
that you build on the ideas we came up with for Boost.Langbinding, but
certainly wouldn't insist on it.

on Thu Aug 25 2011, Stefan Seefeld <stefan-AT-seefeld.name> wrote:

>
> Jim,
>
> this is an interesting idea. There has been lots of general (dare I
> say generic ?) discussion concerning process improvements (which
> unfortunately most of the time diverted into tool discussions). Among
> the fundamental issues is a modularization of boost.  I think it would
> be great if boost.python could follow through on its own, by becoming
> a separate entity. 

Separate from Boost?  I guess that's a possibility but I'm not sure I
see the advantage.

on Thu Aug 25 2011, Ralf Grosse-Kunstleve <rwgrosse-kunstleve-AT-lbl.gov> wrote:

> Hi Jim,
>
> CC to Dave.
>
> This is great news.
> My main interests have been stability and not increasing the memory
> footprint of boost.python extensions. I'm not in a position to further
> develop boost.python.
> Troy and Ravi have done a significant amount of work. 

Yes, and hopefully integrating that could be part of any next steps.

> I hope they will comment for themselves.
> 
> I'd prefer if developments stayed under the boost umbrella, e.g. as
> boost/python/v3, but I don't feel very strongly about this.

Me too.  We managed a transition from v1 to v2 within Boost, and I think
we could do the same for v3.

> On 08/26/2011 07:17 AM, Neal Becker wrote:
>> What sort of improvements did you have in mind?
>
> Two things on my list that are likely going to be somewhat disruptive are:
>
> * Support for subclassing boost.python's own metaclass.

Cool.

> 
> * A per-module type registry, to avoid conflicting converters in
> multi-module projects.

Interesting idea.  How does sharing types across multiple modules work
in that scenario?

on Fri Aug 26 2011, "Niall Douglas" <s_sourceforge-AT-nedprod.com> wrote:

> About a year ago I promised someone in here I'd refactor the BPL 
> interface to native code to support calling a list of functors every 
> time BPL goes in or out of native code. This would allow the Python 
> GIL to be released or reacquired, or indeed many other useful things 
> e.g. benchmarking.

Sounds useful.

on Fri Aug 26 2011, Jim Bosch <talljimbo-AT-gmail.com> wrote:

> On 08/26/2011 04:17 AM, Neal Becker wrote:
>> What sort of improvements did you have in mind?
>
> My list includes:
>
> - Propagating constness to Python (essentially already done as an
> extension, but it could have a much nicer interface if I could mess
> with class_ itself).

Oooh, now that one is tricky.  I'd like to see the design you have in
mind.  Python doesn't have constness; it has immutability, which is
subtly different.

> - Make custom registry and template-based conversions more
> accessible. 

+1

> The former may just need more documentation, but the rvalue converters
> in particular don't seem to have been intended as part of the public
> API originally, and I think they're an important part of the
> library. Template-based conversions are even more buried in the
> details - you have to specialize five or six classes to get it
> working.  I'd like to make it possible to create a template-based
> conversion by (partial) specializing just one traits class.

Hmm, well, IIRC anything you do by pure partial specialization does not
go in the registry and can't participate in cross-module communication,
as it's entirely static.  I guess we need a more formal separation of the
two basic techniques for conversion, not to mention documentation, so
people know to what they're opting in.

> - Automatic conversions for newer boost libraries (Fusion, Pointer
> Container, and Filesystem are at the top of my list), and more for the
> STL and iostreams standard libraries.  I'd like to integrate the
> indexing suite (v2) into Boost.Python proper.

Interesting and useful improvements.

> - Allow Boost.Python wrapped classes to inherit from Python classes.

+1

> - An actual "boost.python" Python module to make exceptions and other
> types used in wrappers easily accessible from Python.

Nice.

> - Some limited degree of priority-based overload matching.  Not sure
> how best to approach this one yet, though.

+1
This is a solved problem... just not in Boost.Python.  Daniel Wallin
worked it out for luabind and we were going to incorporate it into
langbinding.  Happy to discuss it further.


-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com