[C++-sig] EXTERNAL: Re: Odd dlopen behavior

[Davidson, Josh]

Neil, great information, but I did track this problem down to a quirk with Py++.

I've had a great deal of trouble finding a reliable way to actually write modules with module_builder.  Originally, I had been using split_module but I've run into several cases where it goes off in the weeds and tries to write files that exceed the maximum file length.  Generally, this occurs when wrapping classes that go nuts on specialization since module_builder uses the name of the class in the file bindings file.  Here is one quick, extreme example where this occurs when trying to wrap members of boost:
File "C:\Users\davidsj2\workspace\SimCommon\src\Python\goes\build\bindings.py", line 325, in _generate
    files = mb.split_module(self._bindingsDir)
  File "c:\Python26\lib\site-packages\pyplusplus\module_builder\boost_python_builder.py", line 375, in split_module
    , encoding=self.encoding)
  File "c:\Python26\lib\site-packages\pyplusplus\file_writers\__init__.py", line 37, in write_multiple_files
    mfs.write()
  File "c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line 406, in write
    self.split_classes()
  File "c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line 307, in split_classes
    map( self.split_class, class_creators )
  File "c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line 294, in split_class
    self.split_class_impl( class_creator )
  File "c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line 268, in split_class_impl
    , self.create_function_code( function_name ) ) )
  File "c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line 61, in write_file
    writer.writer_t.write_file( fpath, content, self.files_sum_repository, self.encoding )
  File "c:\Python26\lib\site-packages\pyplusplus\file_writers\writer.py", line 150, in write_file
    f = codecs.open( fpath, 'w+b', encoding )
  File "c:\Python26\lib\codecs.py", line 881, in open
    file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory: 'C:\\Users\\davidsj2\\workspace\\SimCommon\\build\\win64\\pybindings\\goes\\boost\\dividable2_less__boost_scope_date_time_scope_date_duration_less__boost_scope_date_time_scope_duration_traits_adapted__greater__comma__int_comma__boost_scope_detail_scope_empty_base_less__boost_scope_date_time_scope_date_duration_less__boost_scope_date_time_scope_duration_traits_adapted__greater___greater___greater_.pypp.hpp'
make: *** [all] Error 1

After finding references to this problem as far back as 2006, I decided to switch over to balanced_split_module.  This has its own set of problems.  The first is that it is highly prone to divide by zero errors.  One quick way to reproduce this issue is to wrap one class and specify a split count of 2.  Obviously not a wise combo, but it's an easy error case that Py++ should handle.

So anyways, the root of *this* problem is how balanced_split_module creates its registration functions.  For each extension, it creates one register function for each file it writes in the form:  void register_classes_<N>()    Obviously, these collide when you create more than one extension using balanced_split_module and enable RTLD_GLOBAL.  One quick solution to this problem would be to prepend the extension name to the name of the registration functions, e.g.: <module>_register_classes_<N>  Since the module name is used to name the files, its easily accessible and would solve a lot of problems.  Of course, if you have modules with the same name in different packages you would run into this again.

Josh

-----Original Message-----
From: cplusplus-sig-bounces+josh.davidson=lmco.com at python.org [mailto:cplusplus-sig-bounces+josh.davidson=lmco.com at python.org] On Behalf Of Niall Douglas
Sent: Wednesday, February 01, 2012 10:46 AM
To: Development of Python/C++ integration
Subject: Re: [C++-sig] EXTERNAL: Re: Odd dlopen behavior

On 31 Jan 2012 at 16:44, Davidson, Josh wrote:

> Ok, well I did figure out the discrepancy between these extensions and 
> previous extensions that have been built that required setting 
> RTLD_GLOBAL.  What I'm doing for these extensions is instead of 
> building in all of the original C++ code AND the Py++ generated code 
> into the extension, I'm only building in the Py++ generated sources  
> and linking an existing shared library containing the original C++ 
> definitions.  Is this non-standard or bad practice?

The big problem with shared objects exporting lots of symbols was that the Linux runtime shared object linker used to have O(N^3) complexity. As a result, every time you ran a program linking to a ginormous shared object you'd get a pause of several seconds as it bound the symbols.

Now, some years ago with KDE and OpenOffice taking forever to load, some eyeballs were turned onto this problem and I know they were going to get it down to O(N^2). There was speak of replacing bits with O(N), but it would introduce ABI compat problems among other things. Another angle was making it use multiple cores. My attention ended up moving elsewhere so I have no idea what has happened since. 
It could still be O(N^2), it could be O(N) or somewhere in between.

> One issue with this is I'm now forced to deliver both the Python 
> extension shared libraries and the original shared libraries.  Not a 
> huge deal, but it does add a little work on the deployment and 
> maintenance end.

On systems with sane DLL designs like Windows and Mac OS X, you'd generally keep the Python bindings separate from the library being bound as it's cleaner and more self-contained. You can also issue smaller self-container ABI compatible releases as hotfixes etc etc.

On the insanity that is ELF, generally you can make inter-SO problems go away by linking everything into a ginormous monolithic SO. However you used to get that O(N^3)/O(N^2) problem I mentioned and maybe you still do. So, sometimes you just have to get your hands dirty and start with hack scripts which post-process the SOs to make their symbol tables sane, or write your own SO loader and binder implementation using dlopen() et al and bypass the system linker altogether :)

Sadly the ISO standards work to enforce sanity in shared libraries across all platforms got dropped from C11 and C++11, but I certainly will try to push that forward again for C11 TR1 along with a few other items on my shopping list (I'm the ISO SC22 convenor for Ireland, though Ireland is only an observer). The problem, as always, is a lack of sponsorship or funding by anyone who cares enough to have the problem fixed properly - and it is a difficult problem to get correct. In the end, as much as these problems are annoying and cost time to people like you, the cost of fixing them isn't seen as business relevant by those with the resources.

HTH,
Niall

--
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 
472909.



_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig at python.org
http://mail.python.org/mailman/listinfo/cplusplus-sig