[C++-sig] Odd dlopen behavior

[Davidson, Josh]

I'm wrapping a C++ project with Py++/Boost.Python under Windows and Linux.  Everything in Windows is working fine, but I'm a bit confused over the behavior in Linux.  The C++ project is built into a single shared library called libsimif, but I'd like to split it up into 3 separate extension modules.  For simplicity, I'll only discuss two of them, since the behavior for the third is identical.  The first, called storage contains definitions of data structures.  It has no dependencies on anything defined in either of the other two extension modules.  The second module, control, uses data structures that are defined in storage.   On the C++ side of things, the headers and source files for storage and control are in entirely different directories.  I've tried a number of different configurations to build the extensions, but one thing that has remained consistent is that for storage, I am only generating Py++ wrappers for the headers included in the storage directory and only building source files in that directory along with the Py++ generated sources.  Ditto for the control extension.

The current configuration that I am using that works passes in libsimif as a library to the distutils.Extension constructor.  Then before starting Python, I need to ensure that libsimif is found in LD_LIBRARY_PATH.  Then I can launch Python and import either module (or from them) and everything works as-expected.   Here is some sample output from this working configuration:
>>> import ast.simif.model_io.storage as storage
>>> import ast.simif.model_io.control as control
>>> dir(storage)
['DiscreteStore', 'PulseStore', 'RtStore', 'SerialStore', 'SharedMemoryBuilder', 'SharedMemoryDeleter', 'SpaceWireStore', '__doc__', '__file__', '__name__', '__package__']
>>> dir(control)
['DiscreteController', 'ModelIoController', 'PulseController', 'RtController', 'SerialController', 'SpaceWireController', '__doc__', '__file__', '__name__', '__package__']
>>> storage.__file__
'ast/simif/model_io/storage.so'
>>> control.__file__
'ast/simif/model_io/control.so'


As you can see, both modules have their own shared library and unique set of symbols.  Now here is why I am confused.  In Linux, we've always set the dlopen flags to include RTLD_NOW and RTLD_GLOBAL.  If I do that, this is what happens:
>>> import sys
>>> import DLFCN
>>> sys.setdlopenflags(DLFCN.RTLD_NOW | DLFCN.RTLD_GLOBAL)
>>> import ast.simif.model_io.storage as storage
>>> import ast.simif.model_io.control as control
__main__:1: RuntimeWarning: to-Python converter for DiscreteStore::FrameData already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for PulseStore::FrameData already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for RtStore::Link already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for RtStore::FrameData already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for RtStore::RtData already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for SerialStore::FrameData already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for SharedMemoryBuilder already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for SharedMemoryDeleter already registered; second conversion method ignored.
>>> dir(storage)
['DiscreteStore', 'PulseStore', 'RtStore', 'SerialStore', 'SharedMemoryBuilder', 'SharedMemoryDeleter', 'SpaceWireStore', '__doc__', '__file__', '__name__', '__package__']
>>> dir(control)
['DiscreteStore', 'PulseStore', 'RtStore', 'SerialStore', 'SharedMemoryBuilder', 'SharedMemoryDeleter', '__doc__', '__file__', '__name__', '__package__']
>>> storage.__file__
'ast/simif/model_io/storage.so'
>>> control.__file__
'ast/simif/model_io/control.so'

So, here storage imports ok, but control complains about a bunch of duplicate registrations.  Then when inspecting the modules, control is completely wrong.  It's like it tried to import storage twice even though __file__ reports the correct shared libraries.   Perhaps not surprising, if  I change the import order and import control ahead of storage, this is what happens:

>>> import sys
>>> import DLFCN
>>> sys.setdlopenflags(DLFCN.RTLD_NOW | DLFCN.RTLD_GLOBAL)
>>> import ast.simif.model_io.control as control
>>> dir(control)
['DiscreteController', 'ModelIoController', 'PulseController', 'RtController', 'SerialController', 'SpaceWireController', '__doc__', '__file__', '__name__', '__package__']
>>> import ast.simif.model_io.storage as storage
__main__:1: RuntimeWarning: to-Python converter for DiscreteController already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for PulseController already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for RtController already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for SerialController already registered; second conversion method ignored.
__main__:1: RuntimeWarning: to-Python converter for SpaceWireController already registered; second conversion method ignored.
>>> dir(storage)
['DiscreteController', 'ModelIoController', 'PulseController', 'RtController', 'SerialController', 'SpaceWireController', 'SpaceWireStore', '__doc__', '__file__', '__name__', '__package__']

Similar behavior, but now the storage import is FUBAR.  Does anyone understand what is going on here?

I'm using x64 Python 2.6.6 on x64 RHEL 6.  Gcc version 4.4.6.

Thanks,
Josh