[C++-sig] Re: sub module support in V2

[Dave Hawkes]

"David Abrahams" <david.abrahams at rcn.com> wrote in message
news:20c801c20e3f$220078e0$6601a8c0 at boostconsulting.com...
>
> From: "Dave Hawkes" <daveh at cadlink.com>
>
> > > All functions are exported on most Unix variants.
> >
> > Though they will only know about each other if they are explicity linked
> to
> > each other. If we have to separate execution units linked to a common
> shared
> > library, they would not be linked to each other.
>
> Believe me; I know what's going on here. I've been involved in long
> discussions with the GCC and GLIBC maintainers about this, as well as some
> with the C++ standards committee about how we're going to standardize
> shared lib behavior.
>

OK, I've written out the module_link in favour of something simpler :)


> > > That's very complicated. There's only one outer-level extension module
> > > being loaded at any given time. Why don't you just keep track of that
> in a
> > > single global which lives in the library.
> >
> > Because if we have multiple extension libraries linked to a common BPL
> then
> > there will be more than one 'global' module as far as the BPL is
> concerned
> > when these extensions are imported by the same python session.
>
> But not at any given instant. Imports are serialized.

Yes, your right, for some reason I was thinking we needed to hang on to it
for the life of the module, but we don't.

>
> > > I'm a little less worried about relying on the ODR than on relying on
a
> > > violation of it. That is, the language standard says "there's one copy
> of
> > > any static data member of a class", and you said "I'm going to write
> code
> > > that only works if there are multiple copies". Of course, you can't
> always
> > > rely on things to work the way the standard says when shared libs are
> > > involved, but relying on things NOT working the way the standard says
> > seems
> > > a bit... perverse to me.
> >
> > Unless the extension library is actually linked to the other this should
> not
> > be a problem, but I think I have a better way of doing it which should
be
> > more robust. I'll test and let you know.
>
> They are linked to a common shared library, so there's a global sharing
> path from one to the other. If the common shared lib instantiates the
> symbol, it will be shared between the two libraries. This doesn't work
> right on all systems today, but that's probably going to change.
>

The new code now doesn't depend on any particular linking envronment.

> > where the BOOST_PYTHON_MODULES() could be located in separate source
> files.
>
> And the advantage to that is...?
>
> (BTW, if you're doing what I think you are, the linker may be free to
> remove the translation units containing the submodule definitions, which
> would break this scheme -- AFAIK, explicit calls are the only portable
> way).
>

I've implemented this for demonstration purposes, and no the linker won't
remove the functions as there addresses are taken by reference in a way that
can't be optimised out.

> > It might not throw a c++ exception, but it can sill throw an access
> > violation, for example, which is caught in catch(...) and possibly saves
> the
> > interpreter from exiting, though may be we don't want that functionality
> > anyway?
>
> Access violations as exceptions are outside the scope of the C++ standard.
> I don't care what happens if there's an access violation -- some invariant
> is broken and the program can't recover reliably anyway. Note the contents
> of libs/python/test/module_tail.cpp which turns these into nice
> JIT-debuggable crashes.
>

In which case there is no need to keep it there, but on the other hand if it
is kept there it has no detrimental effect and may be useful if something is
changed in the future that may create the potential of exceptions at this
point.


> > > What SEH exceptions are continuable?
> >
> > Usually floating point exceptions where we use the MS _fpieee_flt
> > functionality. Typically done in CAD packages that heavily use floating
> > point so we can correct minor overflow/underflow/Inexact problems
without
> > aborting the program. Of course libraries with catch(...) handlers are a
> > menace in this case as it prevents the outer floating point SEH block
> from
> > catching the exception.
>
> More SEH perversity. Is there a way to get the SEH resumption to happen at
> a lower level?
>

Not easily I suspect, without all sorts of rather inconvenient hooks to get
an inner try-except to do what the user wants.

>
> > Unknown c++ exceptions would have to be dealt with by the user as
> > well, but anyone doing this in the first place should be more than
> capable
> > of dealing with that minor problem.
>
> Not sure I agree.
>

If I think of a solution that will not impair the masses, I'll let you
know...

> > Is it a good idea to rely on there being a single common shared library
> for
> > allocation? What if the user overloads the global operator new function
> in
> > the extension library for customised memory allocation?
>
> OK, I hear ya. Really we should use Python's allocator/deallocator for
> these objects, so it's a non-issue.
> For now, the easy answer is to replace "new objects::function(" in
> make_function.hpp with "objects::function::create(...", i.e. a static
> member function call into the library.
>

I agree that would be more appropriate.


I've also implemented some code that allows functions and classes to be
added without chaining so we can do this:

BOOST_PYTHON_MODULE_INIT(PyTest)
{
    boost::python::def("Test1", Test1);

boost::python::add(boost::python::class_<ctest>("ctest").def_init().def("set
_v", &ctest::set_v).def("get_v", &ctest::get_v));
    boost::python::module(".sm");
    boost::python::def("Test2", Test2);
    boost::python::module("PyTest.sm.sm2");
    boost::python::def("Test3", Test3);
}

A similar rationale could probably be applied to classes as you suggested
earlier, but I haven't tried that yet.

Now I was going to send you a diff, but the current CVS appears to be broken
when I include module.hpp...

h:\projects\boost\boost\python\object\make_holder.hpp(54) : error C2059:
syntax error : 'constant'
        h:\projects\boost\boost\python\object\make_holder.hpp(54) : see
reference to class template instantiation
'boost::python::objects::make_holder<16>::apply<Holder,ArgList>' being
compiled
etc.

Dave Hawkes