[C++-sig] [Boost.Python v3] Conversions and Registries

[Niall Douglas]

On 19 Sep 2011 at 17:03, Jim Bosch wrote:

> I'd like to see support for static, template-based conversions.  These 
> would be defined by [partial-]specializing a traits class, and I tend to 
> think they should only be invoked after attempting all registry-based 
> conversions. 

Surely not! You'd want to let template specialisaton be the first 
point of call so the compiler can compile in obvious conversions, 
*then* and only then do you go to a runtime registry.

This also lets one override the runtime registry when needed in the 
local compiland. I'm not against having another set of template 
specialisations do something should the first set of specialisations 
fail, and/or the runtime registry lookup fails.

> Users would have to include the same headers in groups of 
> interdependent modules to avoid specializing the same traits class 
> multiple times in different ways; I can't think of a way to protect them 
> from this, but template-based specializations are a sufficiently 
> advanced featured that I'm comfortable leaving it up to users to avoid 
> this problem.

Just make sure what you do works with precompiled headers :)

P.S.: This is trickier than it sounds.

> We've had some discussion of allowing different modules to have 
> different registries (in addition to a global registry shared across 
> modules).  Leaving aside implementation questions, I have a little 
> survey for interested parties:
> 
> 1) Under what circumstances would you want a conversion that is 
> completely limited to a specific module (or a group of modules that 
> explicitly declare it)?

Defaults to most recent in calling thread stack, but overridable 
using a TLS override to allow impersonation.

The same mechanism usefully also takes care of multiple python 
interpreters too.

> 2) Under what circumstances would you want a conversion to look in a 
> module-specific registry, and then fall back to the global registry?

As above. That implies that there is no global registry, just the 
default registry which all module registries inherit.

> 3) Considering that we will have a "best-match" overloading system, what 
> should take precedence, an inexact match in a module-specific registry, 
> or an exact match in a global registry?  (Clearly this is a moot point 
> for to-Python conversion).

The way I've always done this is to have the template metaprogramming 
set a series of type comparison functions which return scores. This 
pushes most of the scoring and weighting into the compiler and the 
compiler will elide any calls into the dynamic registry where the 
scoring makes that sensible. Makes compile times rather longer though 
:)

The dynamic and compile-time registries can be merged easily enough, 
so all the runtime registry is is a set of comparison functions 
normally elided by the compiler in other modules. In other words, 
mark the inline functions as visible outside the current DLL 
(dllexport/visibility(default)) so the compiler will assemble 
complete versions for external usage.

> Finally, can anyone give me a reason why having a global registry can 
> lead to a violation of the "One Definition Rule"?  This was alluded to 
> many times in the earlier discussion, and there's no doubt that a global 
> registry may lead to unexpected (from a given module's perspective) 
> behavior - but I do not understand the implication that the global 
> registry can result in formally undefined behavior by violating the ODR.

ODR only matters in practice for anything visible outside the current 
compiland. If compiling with GCC -fvisibility=hidden, or on any MSVC 
by default, you can define class foo to be anything you like so long 
as nothing outside the current compiland can see class foo.

ODR is real important though across DLLs. If a DLL X says that class 
foo is one thing and DLL Y says it's something different, expect 
things to go very badly wrong. Hence I simply wouldn't have a global 
registry. It's bad design. You *have* to have per module registries 
and *only* per module registries.

Imagine the following. Program A loads DLL B and DLL C. DLL B is 
dependent on DLL D which uses BPL. DLL C is dependent on DLL E which 
uses BPL.

DLL D tells BPL that class foo is implicitly convertible with an 
integer.

DLL E tells BPL that class foo is actually a thin wrapper for 
std::string.

Right now with present BPL, we have to load two copies of BPL, one 
for DLL D and one for DLL E. They maintain separate type registries, 
so all is good.

But what if DLL B returns a python function to Program A, which then 
installs it as a callback with DLL C?

In the normal case, BPL code in DLL E will call into BPL code DLL D 
and all is well.

But what if the function in DLL D throws an exception?

This gets converted into a C++ exception by throwing 
boost::error_already_set.

Now the C++ runtime must figure where to send the exception. But what 
is the C++ runtime supposed to do with such an exception type? It 
isn't allowed to see the copy of BPL living in DLL E, so it will fire 
the exception type into DLL D where it doesn't belong. At this point, 
the program will almost certainly segfault.

Whatever you do with BPL in the future, it MUST support being a 
dependency of multiple DLLs simultaneously. It MUST know who is 
calling what and when, and know how to unwind everything at any 
particular stage. This implies that it must be 100% compatible with 
dlopen(RTLD_GLOBAL).

As I mentioned earlier, this is a very semantically similar problem 
to supporting multiple python interpreters anyway with each calling 
into one another. You can kill two birds with the one stone as a 
result.

HTH,
Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 
472909.