From sybren at stuvel.eu  Tue May  1 14:43:54 2012
From: sybren at stuvel.eu (=?UTF-8?Q?Sybren_A=2E_St=C3=BCvel?=)
Date: Tue, 1 May 2012 14:43:54 +0200
Subject: [C++-sig] Boost::Python: inheritance + optional parameters
In-Reply-To: <4F9AB87D.7070404@gmail.com>
References: <CAAAfPCCVzw4YxN=QqnfyzJ3xuVGHVU6KOk6scQdtgRMaveJMOQ@mail.gmail.com>
	<4F9AB87D.7070404@gmail.com>
Message-ID: <CAAAfPCDAiwJGOtYi6bKbg_jOJgZBuv8-Lz=jSZESM0pzJHF=9g@mail.gmail.com>

On 27 April 2012 17:17, Jim Bosch <talljimbo at gmail.com> wrote:

> Have you tried using the keyword argument approach to handling default
> arguments?  More precisely, something like this should work, I think:
>
> def("override", func, (arg("x"), arg("y")=0, arg("z")=false))
>
> (where func is the usual wrapper override function).
>

It does, nearly, and I tweaked it to:

def("override", WrapperClass::func, WrapperClass::default_func, (arg("x"),
arg("y")=0, arg("z")=false))

Thank you for your quick answer, you have helped me a lot.

Kind regards,
-- 
Sybren A. St?vel

http://stuvel.eu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120501/6df22d8e/attachment.html>

From s_sourceforge at nedprod.com  Tue May  1 15:18:48 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Tue, 01 May 2012 14:18:48 +0100
Subject: [C++-sig] Wrapping and Passing HWND with Boost.Python
In-Reply-To: <CAOUfC5CVTg_5Moxbsm2s6iQJdC8+d2kXX4T8JZ+yS4zYGA6G5Q@mail.gmail.com>
References: <CAOUfC5BsWxg3WHHGuNSZkXqhFHjChHPzjUHsTUB-jB7khCk1tQ@mail.gmail.com>,
	<CABYYnMy_TUeEF63O-rsA7Gr7qh5=kFtrqVTGbuS-G+W8sJQ8xw@mail.gmail.com>,
	<CAOUfC5CVTg_5Moxbsm2s6iQJdC8+d2kXX4T8JZ+yS4zYGA6G5Q@mail.gmail.com>
Message-ID: <4F9FE2B8.12060.21E3E982@s_sourceforge.nedprod.com>

A HWND is always a void * i.e. an opaque pointer. Unfortunately some 
people using BPL think that BPL can't handle opaque pointers, so they 
do fun stuff like use a thunk struct type for void * instead, and 
wrap the lot in manual pointer casting. It's far easier just to 
declare it an opaque pointer and be done with it.

It is very unfortunate indeed that win32gui returns a HWND as an 
integer. It isn't an integer and must not be treated as one. Still, 
it's easy to write a routine which converts python integers to void 
*.

Niall


On 30 Apr 2012 at 14:54, Ehsan Pi wrote:

> Thanks Roman for your quick reply,
> 
> That's what confuses me. HWND is essentially a pointer to an int to hold
> the window handle. How can to convert them to each other?
> 
> Ehsan
> 
> On Mon, Apr 30, 2012 at 2:46 PM, Roman Yakovenko
> <roman.yakovenko at gmail.com>wrote:
> 
> > On Mon, Apr 30, 2012 at 9:11 PM, Ehsan Pi <ehsanpii at gmail.com> wrote:
> > > In python I get the not-match error:
> > >
> > >     >>> import MyWrapper
> > >     >>> import win32gui
> > >     >>> hwnd = win32gui.GetDesktopWindow()
> > >     >>> foo = MyWrapper.Foo(hwnd)
> > >     Traceback (most recent call last):
> > >       File "<stdin>", line 1, in <module>
> > >     Boost.Python.ArgumentError: Python argument types in
> > >         Foo.__init__(Foo, int)
> > >     did not match C++ signature:
> > >         __init__(struct _object *, struct HWND__ * window)
> > >     >>>
> > >
> > > How can I correct this problem to be able to pass a window's handle (from
> > > win32gui) in Python to C++ class, and interact with it?
> >
> > I think the error is pretty clear: the exported constructor expects a
> > pointer to some struct(?) HWND__, while, win32gui returns the handle
> > as integer. You will have to find some way to associate handle as
> > integer with handle as HWND__.  Once you find it, you can use
> > "make_constructor" functionality, which is also supported by py++.
> >
> > HTH.
> > _______________________________________________
> > Cplusplus-sig mailing list
> > Cplusplus-sig at python.org
> > http://mail.python.org/mailman/listinfo/cplusplus-sig
> >
> 


-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From ehsanpii at gmail.com  Tue May  1 16:21:50 2012
From: ehsanpii at gmail.com (Ehsan Pi)
Date: Tue, 1 May 2012 10:21:50 -0400
Subject: [C++-sig] Wrapping and Passing HWND with Boost.Python
In-Reply-To: <4F9FE2B8.12060.21E3E982@s_sourceforge.nedprod.com>
References: <CAOUfC5BsWxg3WHHGuNSZkXqhFHjChHPzjUHsTUB-jB7khCk1tQ@mail.gmail.com>
	<CABYYnMy_TUeEF63O-rsA7Gr7qh5=kFtrqVTGbuS-G+W8sJQ8xw@mail.gmail.com>
	<CAOUfC5CVTg_5Moxbsm2s6iQJdC8+d2kXX4T8JZ+yS4zYGA6G5Q@mail.gmail.com>
	<4F9FE2B8.12060.21E3E982@s_sourceforge.nedprod.com>
Message-ID: <CAOUfC5AHtDTSEafNfYzG62RyKek4VUG1RiQb-RaOjwz_3rGdBw@mail.gmail.com>

Thanks Niall for your response and correction on HWND.
I'm new to Python (and Boost.Python for that matter) and thought Python was
not typed. If not, how can one convert Python integers to void*? Googled
but didn't find an answer.

Regards,
Ehsan


On Tue, May 1, 2012 at 9:18 AM, Niall Douglas <s_sourceforge at nedprod.com>wrote:

> A HWND is always a void * i.e. an opaque pointer. Unfortunately some
> people using BPL think that BPL can't handle opaque pointers, so they
> do fun stuff like use a thunk struct type for void * instead, and
> wrap the lot in manual pointer casting. It's far easier just to
> declare it an opaque pointer and be done with it.
>
> It is very unfortunate indeed that win32gui returns a HWND as an
> integer. It isn't an integer and must not be treated as one. Still,
> it's easy to write a routine which converts python integers to void
> *.
>
> Niall
>
>
> On 30 Apr 2012 at 14:54, Ehsan Pi wrote:
>
> > Thanks Roman for your quick reply,
> >
> > That's what confuses me. HWND is essentially a pointer to an int to hold
> > the window handle. How can to convert them to each other?
> >
> > Ehsan
> >
> > On Mon, Apr 30, 2012 at 2:46 PM, Roman Yakovenko
> > <roman.yakovenko at gmail.com>wrote:
> >
> > > On Mon, Apr 30, 2012 at 9:11 PM, Ehsan Pi <ehsanpii at gmail.com> wrote:
> > > > In python I get the not-match error:
> > > >
> > > >     >>> import MyWrapper
> > > >     >>> import win32gui
> > > >     >>> hwnd = win32gui.GetDesktopWindow()
> > > >     >>> foo = MyWrapper.Foo(hwnd)
> > > >     Traceback (most recent call last):
> > > >       File "<stdin>", line 1, in <module>
> > > >     Boost.Python.ArgumentError: Python argument types in
> > > >         Foo.__init__(Foo, int)
> > > >     did not match C++ signature:
> > > >         __init__(struct _object *, struct HWND__ * window)
> > > >     >>>
> > > >
> > > > How can I correct this problem to be able to pass a window's handle
> (from
> > > > win32gui) in Python to C++ class, and interact with it?
> > >
> > > I think the error is pretty clear: the exported constructor expects a
> > > pointer to some struct(?) HWND__, while, win32gui returns the handle
> > > as integer. You will have to find some way to associate handle as
> > > integer with handle as HWND__.  Once you find it, you can use
> > > "make_constructor" functionality, which is also supported by py++.
> > >
> > > HTH.
> > > _______________________________________________
> > > Cplusplus-sig mailing list
> > > Cplusplus-sig at python.org
> > > http://mail.python.org/mailman/listinfo/cplusplus-sig
> > >
> >
>
>
> --
> Technology & Consulting Services - ned Productions Limited.
> http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
> Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120501/a4a733d3/attachment.html>

From ehsanpii at gmail.com  Tue May  1 16:25:11 2012
From: ehsanpii at gmail.com (Ehsan Pi)
Date: Tue, 1 May 2012 10:25:11 -0400
Subject: [C++-sig] Wrapping and Passing HWND with Boost.Python
In-Reply-To: <CABYYnMyFAqTOQhKGZ4W1xcQ8jJTO1W26YnbMiySUYYuok-vPNA@mail.gmail.com>
References: <CAOUfC5BsWxg3WHHGuNSZkXqhFHjChHPzjUHsTUB-jB7khCk1tQ@mail.gmail.com>
	<CABYYnMy_TUeEF63O-rsA7Gr7qh5=kFtrqVTGbuS-G+W8sJQ8xw@mail.gmail.com>
	<CAOUfC5CVTg_5Moxbsm2s6iQJdC8+d2kXX4T8JZ+yS4zYGA6G5Q@mail.gmail.com>
	<CABYYnMyFAqTOQhKGZ4W1xcQ8jJTO1W26YnbMiySUYYuok-vPNA@mail.gmail.com>
Message-ID: <CAOUfC5DHXDTOuAJeUvGA2ad+zTVg_iE6TDah=4DtACz6vj8sgg@mail.gmail.com>

Thanks Roman,

I believe these are not specific to py++ but boost.python, correct? What I
mean is whether it can be automated through the py++ wrapper, or do I have
to manually alter the py++ generated cpp file?

Regards,
Ehsan

On Mon, Apr 30, 2012 at 3:00 PM, Roman Yakovenko
<roman.yakovenko at gmail.com>wrote:

> On Mon, Apr 30, 2012 at 9:54 PM, Ehsan Pi <ehsanpii at gmail.com> wrote:
> > Thanks Roman for your quick reply,
> >
> > That's what confuses me. HWND is essentially a pointer to an int to hold
> the
> > window handle. How can to convert them to each other?
>
> I am not sure that you are right, but in case you are, use
> make_constructor and reinterpret_cast.
>
> Regards,
> Roman
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120501/1eb584de/attachment-0001.html>

From roman.yakovenko at gmail.com  Tue May  1 16:37:49 2012
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Tue, 1 May 2012 17:37:49 +0300
Subject: [C++-sig] Wrapping and Passing HWND with Boost.Python
In-Reply-To: <CAOUfC5DHXDTOuAJeUvGA2ad+zTVg_iE6TDah=4DtACz6vj8sgg@mail.gmail.com>
References: <CAOUfC5BsWxg3WHHGuNSZkXqhFHjChHPzjUHsTUB-jB7khCk1tQ@mail.gmail.com>
	<CABYYnMy_TUeEF63O-rsA7Gr7qh5=kFtrqVTGbuS-G+W8sJQ8xw@mail.gmail.com>
	<CAOUfC5CVTg_5Moxbsm2s6iQJdC8+d2kXX4T8JZ+yS4zYGA6G5Q@mail.gmail.com>
	<CABYYnMyFAqTOQhKGZ4W1xcQ8jJTO1W26YnbMiySUYYuok-vPNA@mail.gmail.com>
	<CAOUfC5DHXDTOuAJeUvGA2ad+zTVg_iE6TDah=4DtACz6vj8sgg@mail.gmail.com>
Message-ID: <CABYYnMwskWwno0mmtLomj_-400mXWExY=4Kcbxv+o5OAGA+KMg@mail.gmail.com>

On Tue, May 1, 2012 at 5:25 PM, Ehsan Pi <ehsanpii at gmail.com> wrote:
> Thanks Roman,
>
> I believe these are not specific to py++ but boost.python, correct?

Yes

> What I
> mean is whether it can be automated through the py++ wrapper, or do I have
> to manually alter the py++ generated cpp file?

Yes. py++ supports opaque pointers and code injection. The
documentation contains explanation for different use cases.

From ehsanpii at gmail.com  Tue May  1 16:46:43 2012
From: ehsanpii at gmail.com (Ehsan Pi)
Date: Tue, 1 May 2012 10:46:43 -0400
Subject: [C++-sig] Wrapping and Passing HWND with Boost.Python
In-Reply-To: <CABYYnMwskWwno0mmtLomj_-400mXWExY=4Kcbxv+o5OAGA+KMg@mail.gmail.com>
References: <CAOUfC5BsWxg3WHHGuNSZkXqhFHjChHPzjUHsTUB-jB7khCk1tQ@mail.gmail.com>
	<CABYYnMy_TUeEF63O-rsA7Gr7qh5=kFtrqVTGbuS-G+W8sJQ8xw@mail.gmail.com>
	<CAOUfC5CVTg_5Moxbsm2s6iQJdC8+d2kXX4T8JZ+yS4zYGA6G5Q@mail.gmail.com>
	<CABYYnMyFAqTOQhKGZ4W1xcQ8jJTO1W26YnbMiySUYYuok-vPNA@mail.gmail.com>
	<CAOUfC5DHXDTOuAJeUvGA2ad+zTVg_iE6TDah=4DtACz6vj8sgg@mail.gmail.com>
	<CABYYnMwskWwno0mmtLomj_-400mXWExY=4Kcbxv+o5OAGA+KMg@mail.gmail.com>
Message-ID: <CAOUfC5DZpmB-KTbfoc60nO_X=tUmfaGenNFcs86McYVn8FRiiw@mail.gmail.com>

Thanks again. I'll give it a try as soon as I get a chance.

On Tue, May 1, 2012 at 10:37 AM, Roman Yakovenko
<roman.yakovenko at gmail.com>wrote:

> On Tue, May 1, 2012 at 5:25 PM, Ehsan Pi <ehsanpii at gmail.com> wrote:
> > Thanks Roman,
> >
> > I believe these are not specific to py++ but boost.python, correct?
>
> Yes
>
> > What I
> > mean is whether it can be automated through the py++ wrapper, or do I
> have
> > to manually alter the py++ generated cpp file?
>
> Yes. py++ supports opaque pointers and code injection. The
> documentation contains explanation for different use cases.
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120501/005c046a/attachment.html>

From s_sourceforge at nedprod.com  Tue May  1 17:15:46 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Tue, 01 May 2012 16:15:46 +0100
Subject: [C++-sig] Wrapping and Passing HWND with Boost.Python
In-Reply-To: <CAOUfC5AHtDTSEafNfYzG62RyKek4VUG1RiQb-RaOjwz_3rGdBw@mail.gmail.com>
References: <CAOUfC5BsWxg3WHHGuNSZkXqhFHjChHPzjUHsTUB-jB7khCk1tQ@mail.gmail.com>,
	<4F9FE2B8.12060.21E3E982@s_sourceforge.nedprod.com>,
	<CAOUfC5AHtDTSEafNfYzG62RyKek4VUG1RiQb-RaOjwz_3rGdBw@mail.gmail.com>
Message-ID: <4F9FFE22.28006.224EFDAD@s_sourceforge.nedprod.com>

Cython is the easiest. But using the C API for this is extremely 
easy. Just pull the integer and cast to size_t and then void *.

Niall

On 1 May 2012 at 10:21, Ehsan Pi wrote:

> Thanks Niall for your response and correction on HWND.
> I'm new to Python (and Boost.Python for that matter) and thought Python was
> not typed. If not, how can one convert Python integers to void*? Googled
> but didn't find an answer.
> 
> Regards,
> Ehsan
> 
> 
> On Tue, May 1, 2012 at 9:18 AM, Niall Douglas <s_sourceforge at nedprod.com>wrote:
> 
> > A HWND is always a void * i.e. an opaque pointer. Unfortunately some
> > people using BPL think that BPL can't handle opaque pointers, so they
> > do fun stuff like use a thunk struct type for void * instead, and
> > wrap the lot in manual pointer casting. It's far easier just to
> > declare it an opaque pointer and be done with it.
> >
> > It is very unfortunate indeed that win32gui returns a HWND as an
> > integer. It isn't an integer and must not be treated as one. Still,
> > it's easy to write a routine which converts python integers to void
> > *.
> >
> > Niall
> >
> >
> > On 30 Apr 2012 at 14:54, Ehsan Pi wrote:
> >
> > > Thanks Roman for your quick reply,
> > >
> > > That's what confuses me. HWND is essentially a pointer to an int to hold
> > > the window handle. How can to convert them to each other?
> > >
> > > Ehsan
> > >
> > > On Mon, Apr 30, 2012 at 2:46 PM, Roman Yakovenko
> > > <roman.yakovenko at gmail.com>wrote:
> > >
> > > > On Mon, Apr 30, 2012 at 9:11 PM, Ehsan Pi <ehsanpii at gmail.com> wrote:
> > > > > In python I get the not-match error:
> > > > >
> > > > >     >>> import MyWrapper
> > > > >     >>> import win32gui
> > > > >     >>> hwnd = win32gui.GetDesktopWindow()
> > > > >     >>> foo = MyWrapper.Foo(hwnd)
> > > > >     Traceback (most recent call last):
> > > > >       File "<stdin>", line 1, in <module>
> > > > >     Boost.Python.ArgumentError: Python argument types in
> > > > >         Foo.__init__(Foo, int)
> > > > >     did not match C++ signature:
> > > > >         __init__(struct _object *, struct HWND__ * window)
> > > > >     >>>
> > > > >
> > > > > How can I correct this problem to be able to pass a window's handle
> > (from
> > > > > win32gui) in Python to C++ class, and interact with it?
> > > >
> > > > I think the error is pretty clear: the exported constructor expects a
> > > > pointer to some struct(?) HWND__, while, win32gui returns the handle
> > > > as integer. You will have to find some way to associate handle as
> > > > integer with handle as HWND__.  Once you find it, you can use
> > > > "make_constructor" functionality, which is also supported by py++.
> > > >
> > > > HTH.
> > > > _______________________________________________
> > > > Cplusplus-sig mailing list
> > > > Cplusplus-sig at python.org
> > > > http://mail.python.org/mailman/listinfo/cplusplus-sig
> > > >
> > >
> >
> >
> > --
> > Technology & Consulting Services - ned Productions Limited.
> > http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
> > Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
> >
> >
> >
> > _______________________________________________
> > Cplusplus-sig mailing list
> > Cplusplus-sig at python.org
> > http://mail.python.org/mailman/listinfo/cplusplus-sig
> >
> 


-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From ehsanpii at gmail.com  Tue May  1 21:53:19 2012
From: ehsanpii at gmail.com (Ehsan Pi)
Date: Tue, 1 May 2012 15:53:19 -0400
Subject: [C++-sig] Boost.Python: How to fill a passed in buffer in Python
Message-ID: <CAOUfC5Bo8a527116rkvDzZEmMGY+osASkYbNtvxXAy4on2PUiQ@mail.gmail.com>

Hello forum,

I have a buffer in C++ that I need to fill in Python. The Address of the
buffer is obtained through the GetAddress method which returns a void
pointer to the buffer address.

    #include <boost/smart_ptr/shared_ptr.hpp>
    class Foo
    {
    public:
        Foo(const unsigned int length)
        {
            m_buffer = boost::shared_ptr< unsigned char >( new unsigned
char[ length ] );
        }

        ~Foo(){}

        void* GetAddress( ) const
        {
            // cast for the sake of this question
            return reinterpret_cast< void* >( m_buffer.get() );
        }

    private:
        boost::shared_ptr< unsigned char > m_buffer;
        Foo();
        Foo(const Foo&);
    };

--------------------
Using Py++ I can generate the Boost.Python wrapper to export the class to
Python as follows:

    #include "boost/python.hpp"
    #include "foo.hpp"

    namespace bp = boost::python;

    BOOST_PYTHON_MODULE(MyWrapper){
        { //::Foo
            typedef bp::class_< Foo, boost::noncopyable > Foo_exposer_t;
            Foo_exposer_t Foo_exposer = Foo_exposer_t( "Foo", bp::init<
unsigned int >(( bp::arg("length") )) );
            bp::scope Foo_scope( Foo_exposer );
            bp::implicitly_convertible< unsigned int const, Foo >();
            { //::Foo::GetAddress

                typedef void * ( ::Foo::*GetAddress_function_type )(  )
const;

                Foo_exposer.def(
                    "GetAddress"
                    , GetAddress_function_type( &::Foo::GetAddress )
                    , bp::return_value_policy< bp::return_opaque_pointer
>() );

            }
        }
    }

--------------------
In Python, the output of the GetAddress is a void * to the memory address:

    >>> import MyWrapper
    >>> foo = MyWrapper.Foo(100)
    >>> address = foo.GetAddress()
    >>> print address
    <void * object at 0x01E200B0>
    >>>

My question is can I fill the buffer in Python, and if so how?

Thanks a lot,
Ehsan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120501/612922ef/attachment.html>

From roman.yakovenko at gmail.com  Tue May  1 22:05:23 2012
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Tue, 1 May 2012 23:05:23 +0300
Subject: [C++-sig] Boost.Python: How to fill a passed in buffer in Python
In-Reply-To: <CAOUfC5Bo8a527116rkvDzZEmMGY+osASkYbNtvxXAy4on2PUiQ@mail.gmail.com>
References: <CAOUfC5Bo8a527116rkvDzZEmMGY+osASkYbNtvxXAy4on2PUiQ@mail.gmail.com>
Message-ID: <CABYYnMxDp2i7mncrGDrqA2ntFWp85J_ZNdyynGd6CPtVpjWzHQ@mail.gmail.com>

On Tue, May 1, 2012 at 10:53 PM, Ehsan Pi <ehsanpii at gmail.com> wrote:
> In Python, the output of the GetAddress is a void * to the memory address:
>
> ? ? >>> import MyWrapper
> ? ? >>> foo = MyWrapper.Foo(100)
> ? ? >>> address = foo.GetAddress()
> ? ? >>> print address
> ? ? <void * object at 0x01E200B0>
> ? ? >>>
>
> My question is can I fill the buffer in Python, and if so how?

Easily: using ctypes module. As in previous cases, search py++
documentation for ctypes, as it produces some integration code.

HTH,
Roman

From adam.preble at gmail.com  Sun May  6 22:01:45 2012
From: adam.preble at gmail.com (Adam Preble)
Date: Sun, 6 May 2012 15:01:45 -0500
Subject: [C++-sig] GIL problems during destruction of a
	boost::python::wrapper
Message-ID: <CAE-m0_k15DcDJYDJmcH9EdU7SGVTSuUJBofo0EhEoWwV=gnNCw@mail.gmail.com>

I'm running into my old friend the GIL when it comes time to destroy an
object I've wrapped using boost::python::wrapper.  The object is being
bandied around as a shared pointer, and it's refcount is ending at the time
I want.  Unfortunately, I get the old "PyThreadState_Get: no current
thread" error.  This wrapped class has a lot of virtual functions that
Python-derived classes have to implement.  In the wrapper my methods fight
for the GIL like this:

void TheWrapper::Stuff(blablabla)
{
PyGILState_STATE gstate = PyGILState_Ensure();
this->get_override("Stuff")( blablabla );
PyGILState_Release(gstate);
}

This works fine.  I am assuming I need to write something like this for the
destructor, but I don't see anything about doing that for the destructor.

This might be the wrong approach, so maybe I'll give a little more
information:
1. The object was created in Python.
2. It implements an interface wrapped from the C++ side (the wrapper as
mentioned above).
3. It gives the object to a container on the C++ side.
4. The C++ side has figured out it doesn't need it anymore.
5. This is happening in a different thread from where the object was first
made.
6. Destruction is happening when the removal code goes out of scope.
7. Once the shared_ptr free code starts, the stack descends into the Python
runtime, and there's nothing else on the C++ side coming back up that could
compete with the GIL.  So this is why I believe it is related to the
destructor.

I have pondered acquiring the GIL after explicitly resetting the
shared_ptr, but not all the implementations of that object are actually
implemented in Python, so that would be something of a hatchet job.  I'd
also rather--if I could help it--encapsulate the GIL acquisition stuff in
the wrapper with everything else, rather than remembering to cooperate with
the GIL in choice places sprinkled around my code.  Any ideas?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120506/b3e39618/attachment.html>

From s_sourceforge at nedprod.com  Mon May  7 16:41:46 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 07 May 2012 15:41:46 +0100
Subject: [C++-sig] GIL problems during destruction of a
	boost::python::wrapper
In-Reply-To: <CAE-m0_k15DcDJYDJmcH9EdU7SGVTSuUJBofo0EhEoWwV=gnNCw@mail.gmail.com>
References: <CAE-m0_k15DcDJYDJmcH9EdU7SGVTSuUJBofo0EhEoWwV=gnNCw@mail.gmail.com>
Message-ID: <4FA7DF2A.26260.41231D29@s_sourceforge.nedprod.com>

Only Python may delete a python managed object. You cannot destroy 
any python object explicitly from the C++ side of things.

Solution: allow the python wrapper and C++ instance being wrapped to 
detach from one another. That way if the C++ object is deleted, you 
zombify the python representation of it, and if python refcounts to 
zero the python side, it can be destroyed and can mark the C++ object 
as no longer visible to the python side.

I think BPL used to have explicit special support for auto-detaching 
if you wrap an auto_ptr<> to a C++ object instance? You still need 
some support code your end of course. I remember using a virtual 
factory function taking and returning a void *, you could use this to 
make any arbitrary wrapped C++ object "appear" in python.

Niall

On 6 May 2012 at 15:01, Adam Preble wrote:

> I'm running into my old friend the GIL when it comes time to destroy an
> object I've wrapped using boost::python::wrapper.  The object is being
> bandied around as a shared pointer, and it's refcount is ending at the time
> I want.  Unfortunately, I get the old "PyThreadState_Get: no current
> thread" error.  This wrapped class has a lot of virtual functions that
> Python-derived classes have to implement.  In the wrapper my methods fight
> for the GIL like this:
> 
> void TheWrapper::Stuff(blablabla)
> {
> PyGILState_STATE gstate = PyGILState_Ensure();
> this->get_override("Stuff")( blablabla );
> PyGILState_Release(gstate);
> }
> 
> This works fine.  I am assuming I need to write something like this for the
> destructor, but I don't see anything about doing that for the destructor.
> 
> This might be the wrong approach, so maybe I'll give a little more
> information:
> 1. The object was created in Python.
> 2. It implements an interface wrapped from the C++ side (the wrapper as
> mentioned above).
> 3. It gives the object to a container on the C++ side.
> 4. The C++ side has figured out it doesn't need it anymore.
> 5. This is happening in a different thread from where the object was first
> made.
> 6. Destruction is happening when the removal code goes out of scope.
> 7. Once the shared_ptr free code starts, the stack descends into the Python
> runtime, and there's nothing else on the C++ side coming back up that could
> compete with the GIL.  So this is why I believe it is related to the
> destructor.
> 
> I have pondered acquiring the GIL after explicitly resetting the
> shared_ptr, but not all the implementations of that object are actually
> implemented in Python, so that would be something of a hatchet job.  I'd
> also rather--if I could help it--encapsulate the GIL acquisition stuff in
> the wrapper with everything else, rather than remembering to cooperate with
> the GIL in choice places sprinkled around my code.  Any ideas?
> 


-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From adam.preble at gmail.com  Mon May  7 17:30:41 2012
From: adam.preble at gmail.com (Adam Preble)
Date: Mon, 7 May 2012 10:30:41 -0500
Subject: [C++-sig] GIL problems during destruction of a
	boost::python::wrapper
In-Reply-To: <4FA7DF2A.26260.41231D29@s_sourceforge.nedprod.com>
References: <CAE-m0_k15DcDJYDJmcH9EdU7SGVTSuUJBofo0EhEoWwV=gnNCw@mail.gmail.com>
	<4FA7DF2A.26260.41231D29@s_sourceforge.nedprod.com>
Message-ID: <CAE-m0_mzMBb-XPE7yim1SpAgP4JN=3D6LtCOxG3+FUhY8MKh-g@mail.gmail.com>

I want to make sure I understand the repercussions.

I understand if I were introducing C++ objects into the Python runtime as
internal references that I would be inviting disaster if I delete them on
the C++ side, but continue to use them on the Python side.  I don't think
either of us were talking about that but I wanted to make sure I understood
the boundaries here.

Now on to shared pointer references.  I am utilizing shared pointers for
this particular circumstance.  The object was created in Python and went
out of scope.  If it weren't for the shared pointer, that would be the end
of it, but the pointer was passed to my C++ runtime and retained in a data
structure for future work.  This kept it alive--as I had desired.  When the
object eventually leaves that structure and the refcount drops to zero, it
moves on to destruction.

It does look to me like Python is trying to take care of it since I
immediately pile up through a bunch of Python runtime functions before I
eventually hit my favorite "no such thread" GIL error in the runtime.

I'm not so sure what to do but I can try to search the distribution based
on what you said in hopes of getting some specifics.  For the time being, I
thought it was a deterministic problem, but like most asynchronous stuff,
it went away the next day on a fresh boot with the code slightly altered.
 What I had done was written an empty destructor for the wrapper, just in
anticipating of filling it in with something here.  I can't imagine that
fixing the problem--I wouldn't know why it would fix it.  I think I'll try
to strip that code out and make it come back.

Meanwhile, I'm trying to work on farming all Python work out to a dedicated
thread, and have all these wrappers just inject commands into a stack on
it.  It looks like if I can keep everything bound to there I won't have
issues like this <knocks on wood>.

On Mon, May 7, 2012 at 9:41 AM, Niall Douglas <s_sourceforge at nedprod.com>wrote:

> Only Python may delete a python managed object. You cannot destroy
> any python object explicitly from the C++ side of things.
>
> Solution: allow the python wrapper and C++ instance being wrapped to
> detach from one another. That way if the C++ object is deleted, you
> zombify the python representation of it, and if python refcounts to
> zero the python side, it can be destroyed and can mark the C++ object
> as no longer visible to the python side.
>
> I think BPL used to have explicit special support for auto-detaching
> if you wrap an auto_ptr<> to a C++ object instance? You still need
> some support code your end of course. I remember using a virtual
> factory function taking and returning a void *, you could use this to
> make any arbitrary wrapped C++ object "appear" in python.
>
> Niall
>
> On 6 May 2012 at 15:01, Adam Preble wrote:
>
> > I'm running into my old friend the GIL when it comes time to destroy an
> > object I've wrapped using boost::python::wrapper.  The object is being
> > bandied around as a shared pointer, and it's refcount is ending at the
> time
> > I want.  Unfortunately, I get the old "PyThreadState_Get: no current
> > thread" error.  This wrapped class has a lot of virtual functions that
> > Python-derived classes have to implement.  In the wrapper my methods
> fight
> > for the GIL like this:
> >
> > void TheWrapper::Stuff(blablabla)
> > {
> > PyGILState_STATE gstate = PyGILState_Ensure();
> > this->get_override("Stuff")( blablabla );
> > PyGILState_Release(gstate);
> > }
> >
> > This works fine.  I am assuming I need to write something like this for
> the
> > destructor, but I don't see anything about doing that for the destructor.
> >
> > This might be the wrong approach, so maybe I'll give a little more
> > information:
> > 1. The object was created in Python.
> > 2. It implements an interface wrapped from the C++ side (the wrapper as
> > mentioned above).
> > 3. It gives the object to a container on the C++ side.
> > 4. The C++ side has figured out it doesn't need it anymore.
> > 5. This is happening in a different thread from where the object was
> first
> > made.
> > 6. Destruction is happening when the removal code goes out of scope.
> > 7. Once the shared_ptr free code starts, the stack descends into the
> Python
> > runtime, and there's nothing else on the C++ side coming back up that
> could
> > compete with the GIL.  So this is why I believe it is related to the
> > destructor.
> >
> > I have pondered acquiring the GIL after explicitly resetting the
> > shared_ptr, but not all the implementations of that object are actually
> > implemented in Python, so that would be something of a hatchet job.  I'd
> > also rather--if I could help it--encapsulate the GIL acquisition stuff in
> > the wrapper with everything else, rather than remembering to cooperate
> with
> > the GIL in choice places sprinkled around my code.  Any ideas?
> >
>
>
> --
> Technology & Consulting Services - ned Productions Limited.
> http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
> Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120507/9ecc4d37/attachment.html>

From s_sourceforge at nedprod.com  Mon May  7 20:28:23 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 07 May 2012 19:28:23 +0100
Subject: [C++-sig] GIL problems during destruction of a
	boost::python::wrapper
In-Reply-To: <CAE-m0_mzMBb-XPE7yim1SpAgP4JN=3D6LtCOxG3+FUhY8MKh-g@mail.gmail.com>
References: <CAE-m0_k15DcDJYDJmcH9EdU7SGVTSuUJBofo0EhEoWwV=gnNCw@mail.gmail.com>,
	<4FA7DF2A.26260.41231D29@s_sourceforge.nedprod.com>,
	<CAE-m0_mzMBb-XPE7yim1SpAgP4JN=3D6LtCOxG3+FUhY8MKh-g@mail.gmail.com>
Message-ID: <4FA81447.22946.41F3C879@s_sourceforge.nedprod.com>

On 7 May 2012 at 10:30, Adam Preble wrote:

> I want to make sure I understand the repercussions.
> 
> I understand if I were introducing C++ objects into the Python runtime as
> internal references that I would be inviting disaster if I delete them on
> the C++ side, but continue to use them on the Python side.  I don't think
> either of us were talking about that but I wanted to make sure I understood
> the boundaries here.

What I'm saying is that the wrapper is one object, and the object 
being wrapped is another, totally separate object. Python can delete 
the wrapper at any time, and C++ can delete the object being wrapped 
at any time. That keeps both happy. For very obvious reasons, if you 
delete the object being wrapped you want to reset the smart pointer 
in the wrapper to zero i.e. zombiefy it. I believe BPL knows to throw 
a python exception if the smart pointer in the wrapper is null.

> Now on to shared pointer references.  I am utilizing shared pointers for
> this particular circumstance.  The object was created in Python and went
> out of scope.  If it weren't for the shared pointer, that would be the end
> of it, but the pointer was passed to my C++ runtime and retained in a data
> structure for future work.  This kept it alive--as I had desired.  When the
> object eventually leaves that structure and the refcount drops to zero, it
> moves on to destruction.

If you want C++ code to have a say in the lifetime of a python 
object, simply hold a ref to it. BPL will decrease the refcount when 
the ref gets destructed, thus destroying the object if that's the 
right thing to do.

Generally, though, you DON'T want to keep python wrappers of C++ 
objects around manually [1]. You keep the C++ object around only. If 
python needs to pythonify it, you should set that up to happen on 
demand.

[1]: The obvious exception is when a refcount toggles between zero 
and one through a call stack, thus causing lots of constructions and 
destructions of python wrapper objects. Here it's wise to manually 
hack the refcount.

> It does look to me like Python is trying to take care of it since I
> immediately pile up through a bunch of Python runtime functions before I
> eventually hit my favorite "no such thread" GIL error in the runtime.
> 
> I'm not so sure what to do but I can try to search the distribution based
> on what you said in hopes of getting some specifics.  For the time being, I
> thought it was a deterministic problem, but like most asynchronous stuff,
> it went away the next day on a fresh boot with the code slightly altered.
>  What I had done was written an empty destructor for the wrapper, just in
> anticipating of filling it in with something here.  I can't imagine that
> fixing the problem--I wouldn't know why it would fix it.  I think I'll try
> to strip that code out and make it come back.

An empty destructor inhibits the default constructor. If you're on an 
older MSVC, I vaguely remember a bug where MSVC failed to call a 
default constructor in some circumstances and a quick way of fixing 
it was to write out the default constructor manually.

> Meanwhile, I'm trying to work on farming all Python work out to a dedicated
> thread, and have all these wrappers just inject commands into a stack on
> it.  It looks like if I can keep everything bound to there I won't have
> issues like this <knocks on wood>.

As BPL is currently designed, generally you only want to use it from 
a single thread and indeed just once in any python interpreter. If 
your C++ is inherently multi-threaded though, it can be painful 
serialising everything e.g. use of threading to implement async i/o.

The GIL isn't really a lock, it's actually the per-thread setting of 
what the current interpreter is. What confuses people is that 
sometimes you must set the current interpreter but with the lock 
unlocked for certain functionality to work right. Then people overdo 
it and turn the lock on too frequently, thus introducing deadlocks, 
or people underdo it and you get GIL not set. You can have the "GIL" 
set in the sense of the current interpreter set, but the lock being 
held by a different thread. If you ever try interworking when each 
thread has its own interpreter you'll become a dab hand at this sort 
of stuff.

I know it's frustrating, and there is very little documentation on 
this. I can promise you that if you keep at it, one day it clicks and 
it all starts to work very nicely. Hopefully, sometime this year I'll 
get the funding in place to implement a proper Boost generic type 
registry and things like manual GIL and interpreter management can go 
the way of the dodo.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From talljimbo at gmail.com  Mon May  7 20:38:31 2012
From: talljimbo at gmail.com (Jim Bosch)
Date: Mon, 07 May 2012 14:38:31 -0400
Subject: [C++-sig] GIL problems during destruction of a
	boost::python::wrapper
In-Reply-To: <CAE-m0_mzMBb-XPE7yim1SpAgP4JN=3D6LtCOxG3+FUhY8MKh-g@mail.gmail.com>
References: <CAE-m0_k15DcDJYDJmcH9EdU7SGVTSuUJBofo0EhEoWwV=gnNCw@mail.gmail.com>
	<4FA7DF2A.26260.41231D29@s_sourceforge.nedprod.com>
	<CAE-m0_mzMBb-XPE7yim1SpAgP4JN=3D6LtCOxG3+FUhY8MKh-g@mail.gmail.com>
Message-ID: <4FA816A7.3040908@gmail.com>

On 05/07/2012 11:30 AM, Adam Preble wrote:
> I want to make sure I understand the repercussions.
>
> I understand if I were introducing C++ objects into the Python runtime as
> internal references that I would be inviting disaster if I delete them on
> the C++ side, but continue to use them on the Python side.  I don't think
> either of us were talking about that but I wanted to make sure I understood
> the boundaries here.
>
> Now on to shared pointer references.  I am utilizing shared pointers for
> this particular circumstance.  The object was created in Python and went
> out of scope.  If it weren't for the shared pointer, that would be the end
> of it, but the pointer was passed to my C++ runtime and retained in a data
> structure for future work.  This kept it alive--as I had desired.  When the
> object eventually leaves that structure and the refcount drops to zero, it
> moves on to destruction.
>
> It does look to me like Python is trying to take care of it since I
> immediately pile up through a bunch of Python runtime functions before I
> eventually hit my favorite "no such thread" GIL error in the runtime.
>

You may want to inspect the shared_ptr's deleter object, if you can 
figure out a way to do that (i.e. see if it is a 
boost::python::converter::shared_ptr_deleter).  It seems possible that 
you aren't (for some reason) getting a "native" C++ shared_ptr, but 
rather one that holds a Python object in its custom deleter, and 
invoking that Python thing's destructor is what's causing your problem.


Jim

From matthew.scouten at gmail.com  Tue May  8 08:14:54 2012
From: matthew.scouten at gmail.com (Matthew Scouten)
Date: Tue, 8 May 2012 01:14:54 -0500
Subject: [C++-sig] How can I make myself usefull?
Message-ID: <4fa8b9df.e539320a.654d.42c9@mx.google.com>

Hello,
	I have recently been laid off from a job where a made a lot of good
use of Boost.Python. I would like to give something back (while incidentally
keeping my skills from getting rusty). I have already been following the
boost-python tag on stackoverflow and answering a lot of question there
(http://stackoverflow.com/users/8508/matthew-scouten). I haven't hacked on
the internals of BP much, but now that I have time.

While I was there, I had some code (now lost to me) that made BP easier to
use.  
Here is some of that I had: 

* A deepcopyable suite, so that any c++ class with an appropriate copy ctor
could be quickly given a __deepcopy__, a __copy__ and a copying __init__,
with a single line.

* A similar compare suite, so that classes with == and < could be given a
full set of comparison operators, with a single line

* 2 function templates, SafePointer2Object and SafeObject2Pointer which
dealt with conversions between bp::objects that might be None and  pointers
to c++ classes that might be NULL

* Simple RAII objects that dealt with acquiring and freeing the GIL around
callbacks on different threads. A similar one for freeing the GIL around a
code block. An idea that maybe this could be a call policy. 

* No_compare_indexing_suite is vector_indexing_suite for classes without ==

I would like to recreate some of these up for inclusion in BP, if you are
interested. Or if there is other work that needs doing....
  


From s_sourceforge at nedprod.com  Tue May  8 16:08:33 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Tue, 08 May 2012 15:08:33 +0100
Subject: [C++-sig] How can I make myself usefull?
In-Reply-To: <4fa8b9df.e539320a.654d.42c9@mx.google.com>
References: <4fa8b9df.e539320a.654d.42c9@mx.google.com>
Message-ID: <4FA928E1.1675.462D38EC@s_sourceforge.nedprod.com>

On 8 May 2012 at 1:14, Matthew Scouten wrote:

> Hello,
> 	I have recently been laid off from a job where a made a lot of good
> use of Boost.Python. I would like to give something back (while incidentally
> keeping my skills from getting rusty). I have already been following the
> boost-python tag on stackoverflow and answering a lot of question there
> (http://stackoverflow.com/users/8508/matthew-scouten). I haven't hacked on
> the internals of BP much, but now that I have time.

Firstly, my commiserations to you on losing your job. Secondly, my 
thanks to you for contributing to the SO community. Too few people 
bother. The fact you have used your misfortune to aid others I think 
is especially commendable.

> While I was there, I had some code (now lost to me) that made BP easier to
> use.  

Almost certainly the original code's copyright would have been a 
problem anyway. Besides, reimplementation tends to be of higher 
quality than first attempt, and you'll need that to pass Boost peer 
review.

> Here is some of that I had: 
> 
> * A deepcopyable suite, so that any c++ class with an appropriate copy ctor
> could be quickly given a __deepcopy__, a __copy__ and a copying __init__,
> with a single line.
> 
> * A similar compare suite, so that classes with == and < could be given a
> full set of comparison operators, with a single line
> 
> * 2 function templates, SafePointer2Object and SafeObject2Pointer which
> dealt with conversions between bp::objects that might be None and  pointers
> to c++ classes that might be NULL
> 
> * Simple RAII objects that dealt with acquiring and freeing the GIL around
> callbacks on different threads. A similar one for freeing the GIL around a
> code block. An idea that maybe this could be a call policy. 

I would need to see the design of this before I could say if it was 
correct or not. It's very easy to have something suitable for 
application code, but not suitable to be inside a library, and 
particularly not in BPL.

> * No_compare_indexing_suite is vector_indexing_suite for classes without ==
> 
> I would like to recreate some of these up for inclusion in BP, if you are
> interested. Or if there is other work that needs doing....

http://mail.python.org/pipermail//cplusplus-sig/2011-September/016160.
html (generic runtime type registry, second paragraph onwards)

:)

No, seriously, if you have the free time and can finance an extended 
period of unemployment while you implement such a signature project, 
I would be highly surprised if employment remains a problem. It would 
be a significant commitment in time though. I'd estimate for myself 
around nine months from start to finish, especially as Boost.Python 
and Boost.Spirit would need porting to the new framework, and their 
authors will be very exacting. It would really stand to you long run 
though. I recently was made a job offer by a North American company 
on the basis of signature work I did a decade ago when I was a far 
less capable programmer :)

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From eudoxos at arcig.cz  Fri May 11 09:26:31 2012
From: eudoxos at arcig.cz (=?UTF-8?B?VsOhY2xhdiDFoG1pbGF1ZXI=?=)
Date: Fri, 11 May 2012 09:26:31 +0200
Subject: [C++-sig] using boost::any::type() with boost::python registry for
 to-python convs?
Message-ID: <4FACBF27.5050309@arcig.cz>

Hello,

I read a few posts about converting boost::any to boost::python::object 
in the archive. None of them seems to mention the possibility of using 
boost::any::type() (which is std::type_info) with boost::python registry 
(which identifies types, if I am not wrong, likewise with std::type_info).

Could I get few lines of code on how to work with the registry in such a 
way, if it is possible at all? (I don't need from-python conversions.)

Cheers,

V?clav

From darkant at gmail.com  Sat May 12 18:22:19 2012
From: darkant at gmail.com (DarkAnt)
Date: Sat, 12 May 2012 12:22:19 -0400
Subject: [C++-sig] Nested Boost::Python dictionaries
Message-ID: <CAOvfjcxXd7tA7SrChOaGXjZwOunQ0=RPd_9Wn2dAE0jEncZbew@mail.gmail.com>

I'm trying to create a boost::python::dict that stores another
boost::python::dict.

int main()
{
	Py_Initialize();
	boost::python::dict parent;
	try{
		parent["child_dict"] =
boost::make_shared<boost::python::dict>(boost::python::dict());
	}
	catch(...){
		PyErr_Print();
	}
	return 0;
}

TypeError: No to_python (by-value) converter found for C++ type: class
boost::shared_ptr<class boost::python::dict>

I was under the impression that boost::shared_ptr had special
treatment in boost::python(that is the library already knew what to do
with it). I'm not quite sure if I'm supposed to write this to_python
converter or if I'm supposed to achieve this in a different manner. If
I do write the to_python converter what's the method of keeping track
of both reference counts?

From talljimbo at gmail.com  Sat May 12 22:03:44 2012
From: talljimbo at gmail.com (Jim Bosch)
Date: Sat, 12 May 2012 16:03:44 -0400
Subject: [C++-sig] Nested Boost::Python dictionaries
In-Reply-To: <CAOvfjcxXd7tA7SrChOaGXjZwOunQ0=RPd_9Wn2dAE0jEncZbew@mail.gmail.com>
References: <CAOvfjcxXd7tA7SrChOaGXjZwOunQ0=RPd_9Wn2dAE0jEncZbew@mail.gmail.com>
Message-ID: <4FAEC220.4030005@gmail.com>

On 05/12/2012 12:22 PM, DarkAnt wrote:
> I'm trying to create a boost::python::dict that stores another
> boost::python::dict.
>
> int main()
> {
> 	Py_Initialize();
> 	boost::python::dict parent;
> 	try{
> 		parent["child_dict"] =
> boost::make_shared<boost::python::dict>(boost::python::dict());
> 	}
> 	catch(...){
> 		PyErr_Print();
> 	}
> 	return 0;
> }
>
> TypeError: No to_python (by-value) converter found for C++ type: class
> boost::shared_ptr<class boost::python::dict>
>
> I was under the impression that boost::shared_ptr had special
> treatment in boost::python(that is the library already knew what to do
> with it). I'm not quite sure if I'm supposed to write this to_python
> converter or if I'm supposed to achieve this in a different manner. If
> I do write the to_python converter what's the method of keeping track
> of both reference counts?

There's no need to use shared_ptr on Python objects; those will be 
tracked using Python's own reference counting system.

Just doing

parent["child_dict"] = boost::python::dict();

will do what you want.

Note that boost::python::dict (as well as boost::python::object, etc.) 
is actually a smart pointer itself, holding a PyObject* and 
incrementing/decrementing the reference count in the C++ copy 
constructor, assignment operator, and destructor.

HTH!

Jim

From cowingzitron at yahoo.com  Sat May 12 22:10:37 2012
From: cowingzitron at yahoo.com (Christopher Cowing-Zitron)
Date: Sat, 12 May 2012 13:10:37 -0700 (PDT)
Subject: [C++-sig] Extensive memory errors upon C++ destruction and data
	transfer from numpy
Message-ID: <1336853437.41351.YahooMailNeo@web160302.mail.bf1.yahoo.com>

Hello,

I am writing a computational package integrating Python and C++ using Boost.Python. The interface and the initial data collection are in Python, but the workhorse data structures and computation are in C++. I am having difficulties with memory errors at two points: at the transfer of data between numpy and C++, and upon the destruction of wrapped C++ objects. I suspect the issues are related, and are not the result of limitations of Python, C++, or Boost.Python, but rather are the result of my lack of familiarity with C++. I apologize for the long email, but my lack of experience with C++ makes it hard to describe my problems succinctly, especially because I have no idea where the problems are occurring, nor do I know the right vocabulary. I've pasted below the C++ source for the relevant class, as well as the Boost.Python module wrapper code. This is all for academic research, and under the legal conditions of my grants it is necessarily open-source,
 but because I intend to publish this package in a research journal, I'd appreciate it if you didn't share the code beyond this mailing list; I'll create pypi and sourceforge entries for the source once it's ready for release. Most importantly, feel free to talk trash about and point out any errors in my code: I just started with C++ three weeks ago, and criticism is very helpful and much appreciated! :-)


My package is for the analysis of large biological data sets, and a new C++ object must be created for each gene analyzed, which for the human genome amounts to about 30,000 times for the analysis of a single data set. A single gene object stores many variables, some of which are scalar and could be built into the class as static variables, but most variables are array data which greatly vary in size (and possibly dimension) depending upon the biological structure of the individual gene as well as the nature of the experiment being analyze, which cannot be known in advance. As such, all the array variables are dynamically allocated. I have used Boost.Python shared pointers and shared arrays for all dynamically allocated variables. In an attempt at uniformity, in this version of the code, all gene class variables are dynamically allocated.

I have separated the creation of a gene object and the filling of variable values into two steps. All access of C++ variables is done through wrapped get/set methods, and all data is transferred via numpy arrays. I have no problems upon object creation, but I get memory errors both when filling the variables from numpy arrays, and transferring calculated variable values back into numpy arrays. The issue seems to be that I cannot force a copy of variable values between numpy arrays and C++, but rather Boost.Python seems intent upon using references (in fact the documentation for the extract<> template explicitly states that, and I can't find any other way to get variables out of numpy arrays and into C++). This matters because the C++ gene object will modify some of its own variable values that were initially taken from Python, and the user will similarly modify within Python variables used to fill the gene object, and I need to make sure changes in one
 do not get automatically reflected in the other. In both cases I get bus errors / segmentation faults.


The second problem relates to object deletion. Once a single gene has been analyzed, neither the C++ object nor the corresponding Python object are needed; the important numerical results are simply stored in a numpy array. Because of the huge number of genes, I have one of two options when it comes to memory management, in order to avoid the build-up of excess memory consumption by genes for which the analysis has been completed: I can either explicitly destroy the C++ object, by brute force or by reassigning a new object to its variable in Python, or I can reinitialize its variables to store a new gene (I am inclined to the second). But whenever I try to destroy a wrapped C++ object, using either method, I get a segmentation fault. I've written a C++ destructor for my gene class; I tried to wrap it for Python, but the Boost.Python documentation doesn't mention how to register destructors to Python, so I just wrapped it like any other bound method.
 (Note: that wrapper is not present in the version I've posted below, because I assumed I got it wrong anyway). I still get the faults using that method, as called by my assigned name. Should I be using the Python object's .__del__ method or the global Python del(), and if so, how do I specialize them to my gene object's destructor? Even if I go with the option of just reinitializing the C++ gene object variables, rather than explicit deletion, I still get a segmentation fault when I leave the Python interpreter, presumably because it's trying to destroy the wrapped C++ gene objects created in that session.

Any and all help and advice is very much appreciated, and I thank you in advance for your help. If any of you provide particular help, I will be happy to include you in the acknowledgements should I get this darn thing published; I'd offer co-authorship, but my boss would never allow it. Have a good weekend.

-- Chris Cowing-Zitron

fp_boosters.hpp, the source file defining the Boost.Python module:


#include <fp_genes.hpp>

BOOST_PYTHON_MODULE(fp_boosters)
{
??? def("loggamma", &fingerpuppet::evaluations::loggamma_py, return_value_policy<return_by_value>());
??? def("digamma", &fingerpuppet::evaluations::digamma_py, return_value_policy<return_by_value>());
??? def("trigamma", &fingerpuppet::evaluations::trigamma_py, return_value_policy<return_by_value>());
??? def("ll_single", &fingerpuppet::evaluations::ll_single_py);
??? def("ll_grad_single", &fingerpuppet::evaluations::ll_grad_single_py);
??? def("ll_hess_single", &fingerpuppet::evaluations::ll_hess_single_py);
??? def("ll_full", &fingerpuppet::evaluations::ll_full_py);
??? def("ll_grad_full", &fingerpuppet::evaluations::ll_grad_full_py);
??? def("ll_hess_full", &fingerpuppet::evaluations::ll_hess_full_py);

??? class_<fingerpuppet::genes::gene, boost::noncopyable>("gene")

??? ??? .def("register_gene", &fingerpuppet::genes::gene::register_new_gene_py)
??? ??? .def("reset_gene", &fingerpuppet::genes::gene::reset_gene)
??? ??? .def("empty_gene", &fingerpuppet::genes::gene::empty_gene)
??? ??? .def("eval_ll", &fingerpuppet::genes::gene::eval_ll_py)
??? ??? .def("eval_cons", &fingerpuppet::genes::gene::eval_cons_py)
??? ??? .def("eval_ll_grad", &fingerpuppet::genes::gene::eval_ll_grad_py)
??? ??? .def("eval_cons_jac", &fingerpuppet::genes::gene::eval_cons_jac_py)
??? ??? .def("eval_ll_hess", &fingerpuppet::genes::gene::eval_ll_hess_py)
??? ??? .def("eval_cons_hess", &fingerpuppet::genes::gene::eval_cons_hess_py)
??? ??? .def("eval_lagrangian", &fingerpuppet::genes::gene::eval_lagrangian_py)
??? ??? .def("update_var", &fingerpuppet::genes::gene::update_var_py)
??? ??? .def("return_var", &fingerpuppet::genes::gene::return_var_py)

??? ;

??? numeric::array::set_module_and_type("numpy", "ndarray");
};




fp_genes.hpp, the header defining the gene object class:

#include <fp_evaluations.hpp>


namespace fingerpuppet {


namespace genes { 

class gene;

class gene : public boost::enable_shared_from_this<gene>, public boost::noncopyable
??? ??? 
{ 


public:


/* Constructors, destructor, and construction utilities */

gene()


{
??? gene_count = 0;
??? reset_gene();
??? 
}


~gene()

{

??? empty_gene();

}


boost::shared_ptr<gene> return_this()

{

??? return shared_from_this();

}


bool register_new_gene(int *ns, int *statecolsin, int *sampcolsin, int *exonrowsin, int *segrowsin, double *readsin)

{

??? reset_gene();
??? gene_count += 1;
??? 
??? nstate.reset(new int);
??? nsamp.reset(new int);
??? nexon.reset(new int);
??? nseg.reset(new int);
??? nwin.reset(new int);
??? nvar.reset(new int);
??? ncon.reset(new int);
??? nsingle.reset(new int);
??? hsingle.reset(new int);
??? jac_entries.reset(new int);
??? hess_entries.reset(new int);
??? current_ll.reset(new double);
??? current_obj_factor.reset(new double);

??? *nstate = ns[0];
??? *nsamp = ns[1];
??? *nexon = ns[2];
??? *nseg = ns[3];
??? *nwin = ns[4];
??? *nvar = ns[5];
??? *ncon = *nsamp * *nexon;
??? *nsingle = 9;
??? *hsingle = 21;
??? *jac_entries = 2 * *ncon;
??? *hess_entries = 0;
??? *current_ll = 0.0;
??? *current_obj_factor = 1.0;
??? 
??? statecols.reset(new int [*nsamp]);
??? sampcols.reset(new int [*nsamp]);
??? exonrows.reset(new int [*nwin]);
??? segrows.reset(new int [*nwin]);
??? sampsperstate.reset(new int [*nstate]);
??? segsperexon.reset(new int [*nseg]);
??? single_hess_rows.reset(new int [*hsingle]);
??? single_hess_cols.reset(new int [*hsingle]);
??? jac_rows.reset(new int [*jac_entries]);
??? jac_cols.reset(new int [*jac_entries]);
??? eval_counts.reset(new int [7]);

??? reads.reset(new double [*nsamp * *nwin]);
??? readmaxes.reset(new double [*nsamp * *nexon]);
??? start_X.reset(new double [*nvar]);
??? x_L.reset(new double [*nvar]);
??? x_U.reset(new double [*nvar]);
??? g_L.reset(new double [*ncon]);
??? g_U.reset(new double [*ncon]);
??? current_X.reset(new double [*nvar]);
??? current_cons.reset(new double [*ncon]);
??? current_ll_grad.reset(new double [*nvar]);
??? current_cons_jac.reset(new double [*jac_entries]);
??? current_cons_hess.reset(new double [*ncon]);
??? current_lambda.reset(new double [*ncon]);

??? 
??? for (int entry = 0; entry < *ncon; ++entry) { 
??? ??? readmaxes[entry] = 0.0;
??? }
??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? for (int win = 0; win < *nwin; ++win) { 
??? ??? ??? if (reads[(samp * *nwin) + win] > readmaxes[(samp * *nexon) + exonrows[win]]) { 
??? ??? ??? ??? readmaxes[(samp * *nexon) + exonrows[win]] = reads[(samp * *nwin) + win];
??? ??? ??? }
??? ??? }
??? }
??? 
??? for (int entry = 0; entry < *nstate; ++entry) { 
??? ??? sampsperstate[entry] = 0;
??? }
??? 
??? for (int entry = 0; entry < *nsamp; ++entry) { 
??? ??? sampsperstate[statecols[entry]] += 1;
??? }
??? 
??? for (int entry = 0; entry < *nexon; ++entry) { 
??? ??? segsperexon[entry] = 0;
??? }
??? 
??? segsperexon[0] = 1;
??? 
??? for (int entry = 1; entry < *nwin; ++entry) { 
??? ??? if (segrows[entry] != segrows[entry - 1]) { 
??? ??? ??? segsperexon[exonrows[entry]] += 1;
??? ??? }
??? }
??? ??? 
??? init_jac_struct();
??? init_hess_struct();
??? init_bounds();
??? init_start_X();


}


bool register_new_gene_py(numeric::array const &ns_py, numeric::array const &statecolsin_py, numeric::array const &sampcolsin_py, numeric::array const &exonrowsin_py, numeric::array const &segrowsin_py, numeric::array const &readsin_py)


{

??? reset_gene();
??? gene_count += 1;

??? nstate.reset(new int);
??? nsamp.reset(new int);
??? nexon.reset(new int);
??? nseg.reset(new int);
??? nwin.reset(new int);
??? nvar.reset(new int);
??? ncon.reset(new int);
??? nsingle.reset(new int);
??? hsingle.reset(new int);
??? jac_entries.reset(new int);
??? hess_entries.reset(new int);
??? current_ll.reset(new double);
??? current_obj_factor.reset(new double);

??? *nstate = int(ns_py[0]);
??? *nsamp = int(ns_py[1]);
??? *nexon = int(ns_py[2]);
??? *nseg = int(ns_py[3]);
??? *nwin = int(ns_py[4]);
??? *nvar = int(ns_py[5]);
??? *ncon = *nsamp * *nexon;
??? *nsingle = 9;
??? *hsingle = 21;
??? *jac_entries = 2 * *ncon;
??? *hess_entries = 0;
??? *current_ll = 0.0;
??? *current_obj_factor = 1.0;

??? statecols.reset(new int [*nsamp]);
??? sampcols.reset(new int [*nsamp]);
??? exonrows.reset(new int [*nwin]);
??? segrows.reset(new int [*nwin]);
??? sampsperstate.reset(new int [*nstate]);
??? segsperexon.reset(new int [*nseg]);
??? single_hess_rows.reset(new int [*hsingle]);
??? single_hess_cols.reset(new int [*hsingle]);
??? jac_rows.reset(new int [*jac_entries]);
??? jac_cols.reset(new int [*jac_entries]);
??? eval_counts.reset(new int [7]);

??? reads.reset(new double [*nsamp * *nwin]);
??? readmaxes.reset(new double [*nsamp * *nexon]);
??? start_X.reset(new double [*nvar]);
??? x_L.reset(new double [*nvar]);
??? x_U.reset(new double [*nvar]);
??? g_L.reset(new double [*ncon]);
??? g_U.reset(new double [*ncon]);
??? current_X.reset(new double [*nvar]);
??? current_cons.reset(new double [*ncon]);
??? current_ll_grad.reset(new double [*nvar]);
??? current_cons_jac.reset(new double [*jac_entries]);
??? current_cons_hess.reset(new double [*ncon]);
??? current_lambda.reset(new double [*ncon]);
??? 
??? for (int entry = 0; entry < *nsamp; ++entry) { 
??? ??? statecols[entry] = extract<int>(statecolsin_py[entry]());
??? }
??? 
??? for (int entry = 0; entry < *nsamp; ++entry) { 
??? ??? sampcols[entry] = extract<int>(sampcolsin_py[entry]());
??? }
??? 
??? for (int entry = 0; entry < *nwin; ++entry) { 
??? ??? exonrows[entry] = extract<int>(exonrowsin_py[entry]());
??? }
??? 
??? for (int entry = 0; entry < *nwin; ++entry) { 
??? ??? segrows[entry] = extract<int>(segrowsin_py[entry]());
??? }
??? 
??? for (int entry = 0; entry < *nsamp * *nwin; ++entry) { 
??? ??? reads[entry] = extract<double>(readsin_py[entry]());
??? }
??? 
??? for (int entry = 0; entry < *ncon; ++entry) { 
??? ??? readmaxes[entry] = 0.0;
??? }
??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? for (int win = 0; win < *nwin; ++win) { 
??? ??? ??? if (reads[(samp * *nwin) + win] > readmaxes[(samp * *nexon) + exonrows[win]]) { 
??? ??? ??? ??? readmaxes[(samp * *nexon) + exonrows[win]] = reads[(samp * *nwin) + win];
??? ??? ??? }
??? ??? }
??? }
??? 
??? for (int entry = 0; entry < *nstate; ++entry) { 
??? ??? sampsperstate[entry] = 0;
??? }
??? 
??? for (int entry = 0; entry < *nsamp; ++entry) { 
??? ??? sampsperstate[statecols[entry]] += 1;
??? }
??? 
??? for (int entry = 0; entry < *nexon; ++entry) { 
??? ??? segsperexon[entry] = 0;
??? }
??? 
??? segsperexon[0] = 1;
??? 
??? for (int entry = 1; entry < *nwin; ++entry) { 
??? ??? if (segrows[entry] != segrows[entry - 1]) { 
??? ??? ??? segsperexon[exonrows[entry]] += 1;
??? ??? }
??? }
??? ??? 
??? init_jac_struct();
??? init_hess_struct();
??? init_bounds();
??? init_start_X();

}

bool reset_gene()

{

??? nstate.reset(new int);
??? nsamp.reset(new int);
??? nexon.reset(new int);
??? nseg.reset(new int);
??? nwin.reset(new int);
??? nvar.reset(new int);
??? ncon.reset(new int);
??? nsingle.reset(new int);
??? hsingle.reset(new int);
??? jac_entries.reset(new int);
??? hess_entries.reset(new int);
??? current_ll.reset(new double);
??? current_obj_factor.reset(new double);
??? statecols.reset(new int [1]);
??? sampcols.reset(new int [1]);
??? exonrows.reset(new int [1]);
??? segrows.reset(new int [1]);
??? sampsperstate.reset(new int [1]);
??? segsperexon.reset(new int [1]);
??? single_hess_rows.reset(new int [1]);
??? single_hess_cols.reset(new int [1]);
??? jac_rows.reset(new int [1]);
??? jac_cols.reset(new int [1]);
??? full_hess_rows.reset(new int [1]);
??? full_hess_cols.reset(new int [1]);
??? eval_counts.reset(new int [1]);
??? reads.reset(new double [1]);
??? readmaxes.reset(new double [1]);
??? start_X.reset(new double [1]);
??? x_L.reset(new double [1]);
??? x_U.reset(new double [1]);
??? g_L.reset(new double [1]);
??? g_U.reset(new double [1]);
??? current_X.reset(new double [1]);
??? current_cons.reset(new double [1]);
??? current_ll_grad.reset(new double [1]);
??? current_cons_jac.reset(new double [1]);
??? current_ll_hess.reset(new double [1]);
??? current_cons_hess.reset(new double [1]);
??? current_lambda.reset(new double [1]);
??? current_lagrangian.reset(new double [1]);
??? 
??? return true;

}


bool empty_gene()

{

??? nstate.reset();
??? nsamp.reset();
??? nexon.reset();
??? nseg.reset();
??? nwin.reset();
??? nvar.reset();
??? ncon.reset();
??? nsingle.reset();
??? hsingle.reset();
??? jac_entries.reset();
??? hess_entries.reset();
??? current_ll.reset();
??? current_obj_factor.reset();
??? statecols.reset();
??? sampcols.reset();
??? exonrows.reset();
??? segrows.reset();
??? sampsperstate.reset();
??? segsperexon.reset();
??? single_hess_rows.reset();
??? single_hess_cols.reset();
??? jac_rows.reset();
??? jac_cols.reset();
??? full_hess_rows.reset();
??? full_hess_cols.reset();
??? eval_counts.reset();
??? reads.reset();
??? readmaxes.reset();
??? start_X.reset();
??? x_L.reset();
??? x_U.reset();
??? g_L.reset();
??? g_U.reset();
??? current_X.reset();
??? current_cons.reset();
??? current_ll_grad.reset();
??? current_cons_jac.reset();
??? current_ll_hess.reset();
??? current_cons_hess.reset();
??? current_lambda.reset();
??? current_lagrangian.reset();
??? return true;

}


bool init_jac_struct()

{
??? ??? ??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? jac_rows[2 * ((samp * *nexon) + exon)] = (6 * *nstate) + samp;
??? ??? ??? jac_cols[2 * ((samp * *nexon) + exon)] = (samp * *nexon) + exon;
??? ??? ??? jac_rows[(2 * ((samp * *nexon) + exon)) + 1] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
??? ??? ??? jac_cols[(2 * ((samp * *nexon) + exon)) + 1] = (samp * *nexon) + exon;
??? ??? }
??? }
??? 
??? return true;
??? 
}


bool init_hess_struct()

{

??? int shr[21] = {3,4,2,4,2,5,6,1,6,1,7,8,0,8,0,0,1,2,1,2,2};
??? int shc[21] = {3,3,3,4,4,5,5,5,6,6,7,7,7,8,8,0,0,0,1,1,2};
??? 
??? for (int entry = 0; entry < 21; ++entry) { 
??? ??? single_hess_rows[entry] = shr[entry];
??? ??? single_hess_cols[entry] = shc[entry];
??? }
??? 
??? for (int state = 0; state < *nstate; ++state) { 
??? ??? *hess_entries += 3 + (2 * *nseg * sampsperstate[state]);
??? }
??? 
??? for (int state = 0; state < *nstate; ++state) { 
??? ??? *hess_entries += 3 + (2 * *nexon * sampsperstate[state]);
??? }
??? 
??? for (int state = 0; state < *nstate; ++state) { 
??? ??? *hess_entries += 3 + (2 * sampsperstate[state]);
??? }
??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? *hess_entries += 1 + *nexon + *nseg;
??? }
??? 
??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? *hess_entries += *nsamp * (1 + segsperexon[exon]);
??? }
??? 
??? *hess_entries += *nsamp * *nseg;
??? ??? 
??? full_hess_rows.reset(new int [*hess_entries]);
??? full_hess_cols.reset(new int [*hess_entries]);
??? current_ll_hess.reset(new double [*hess_entries]);
??? current_lagrangian.reset(new double [*hess_entries]);

??? int entry = 0;
??? 
??? for (int state = 0; state < *nstate; ++state) {
??? ??? for (int var = 0; var < 2 + (*nseg * sampsperstate[state]); ++var) { 
??? ??? ??? full_hess_cols[entry + var] = state;
??? ??? }
??? ??? full_hess_rows[entry] = state;
??? ??? full_hess_rows[entry + 1] = *nstate + state;
??? ??? entry += 2;
??? ??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? ??? for (int seg = 0; seg < *nseg; ++seg) { 
??? ??? ??? ??? if (statecols[samp] == state) { 
??? ??? ??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + (*nsamp * (*nexon + 1)) + (samp * *nseg) + seg;
??? ??? ??? ??? ??? entry += 1;
??? ??? ??? ??? }
??? ??? ??? }
??? ??? }
??? }

??? for (int state = 0; state < *nstate; ++state) {
??? ??? for (int var = 0; var < 1 + (*nseg * sampsperstate[state]); ++var) { 
??? ??? ??? full_hess_cols[entry + var] = *nstate + state;
??? ??? }
??? ??? full_hess_rows[entry] = *nstate + state;
??? ??? entry += 1;
??? ??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? ??? for (int seg = 0; seg < *nseg; ++seg) { 
??? ??? ??? ??? if (statecols[samp] == state) { 
??? ??? ??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + (*nsamp * (*nexon + 1)) + (samp * *nseg) + seg;
??? ??? ??? ??? ??? entry += 1;
??? ??? ??? ??? }
??? ??? ??? }
??? ??? }
??? }
??? 
??? for (int state = 0; state < *nstate; ++state) {
??? ??? for (int var = 0; var < 2 + (*nexon * sampsperstate[state]); ++var) { 
??? ??? ??? full_hess_cols[entry + var] = (2 * *nstate) + state;
??? ??? }
??? ??? full_hess_rows[entry] = (2 * *nstate) + state;
??? ??? full_hess_rows[entry + 1] = (3 * *nstate) + state;
??? ??? entry += 2;
??? ??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? ??? if (statecols[samp] == state) { 
??? ??? ??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
??? ??? ??? ??? ??? entry += 1;
??? ??? ??? ??? }
??? ??? ??? }
??? ??? }
??? }

??? for (int state = 0; state < *nstate; ++state) {
??? ??? for (int var = 0; var < 1 + (*nexon * sampsperstate[state]); ++var) { 
??? ??? ??? full_hess_cols[entry + var] = (3 * *nstate) + state;
??? ??? }
??? ??? full_hess_rows[entry] = (3 * *nstate) + state;
??? ??? entry += 1;
??? ??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? ??? if (statecols[samp] == state) { 
??? ??? ??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
??? ??? ??? ??? ??? entry += 1;
??? ??? ??? ??? }
??? ??? ??? }
??? ??? }
??? }
??? 
??? for (int state = 0; state < *nstate; ++state) {
??? ??? for (int var = 0; var < 2 + sampsperstate[state]; ++var) { 
??? ??? ??? full_hess_cols[entry + var] = (4 * *nstate) + state;
??? ??? }
??? ??? full_hess_rows[entry] = (4 * *nstate) + state;
??? ??? full_hess_rows[entry + 1] = (5 * *nstate) + state;
??? ??? entry += 2;
??? ??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? ??? if (statecols[samp] == state) { 
??? ??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + samp;
??? ??? ??? ??? entry += 1;
??? ??? ??? }
??? ??? }
??? }

??? for (int state = 0; state < *nstate; ++state) {
??? ??? for (int var = 0; var < 1 + sampsperstate[state]; ++var) { 
??? ??? ??? full_hess_cols[entry + var] = (5 * *nstate) + state;
??? ??? }
??? ??? full_hess_rows[entry] = (5 * *nstate) + state;
??? ??? entry += 1;
??? ??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? ??? if (statecols[samp] == state) { 
??? ??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + samp;
??? ??? ??? ??? entry += 1;
??? ??? ??? }
??? ??? }
??? }
??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? for (int var = 0; var < 1 + *nexon + *nseg; ++var) { 
??? ??? ??? full_hess_cols[entry + var] = (6 * *nstate) + samp;
??? ??? }
??? ??? full_hess_rows[entry] = (6 * *nstate) + samp;
??? ??? entry += 1;
??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
??? ??? ??? entry += 1;
??? ??? }
??? ??? for (int seg = 0; seg < *nseg; ++seg) { 
??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + (*nsamp * (*nexon + 1)) + (samp * *nseg) + seg;
??? ??? ??? entry += 1;
??? ??? }
??? }
??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? int startseg = 0;
??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? for (int var = 0; var < 1 + segsperexon[exon]; ++var) { 
??? ??? ??? ??? full_hess_cols[entry + var] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
??? ??? ??? }
??? ??? ??? startseg += segsperexon[exon];
??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
??? ??? ??? entry += 1;
??? ??? ??? for (int seg = startseg; seg < startseg + segsperexon[exon]; ++seg) { 
??? ??? ??? ??? full_hess_rows[entry] = (6 * *nstate) + (*nsamp * (*nexon + 1)) + (samp * *nseg) + seg;
??? ??? ??? ??? entry += 1;
??? ??? ??? }
??? ??? }
??? }
??? 
??? for (int var = (6 * *nstate) + (*nsamp * (*nexon + 1)); var < *nvar; ++var) {
??? ??? full_hess_cols[entry] = var;
??? ??? full_hess_rows[entry] = var;
??? ??? entry += 1;
??? }??? 
??? 
??? return true;

} 


bool init_bounds()

{

??? double curmax;
??? ??? 
??? for (int entry = 0; entry < 4 * *nvar; ++entry) { 
??? ??? x_L[entry] = 0.01;
??? ??? x_U[entry] = 100.0;
??? }
??? 
??? for (int entry = (6 * *nstate) + *nsamp; entry < *nvar; ++entry) { 
??? ??? x_L[entry] = 0.001;
??? ??? x_U[entry] = 0.999;
??? }
??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? curmax = 0;
??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? if (readmaxes[(samp * *nexon) + exon] > curmax) { 
??? ??? ??? ??? curmax = readmaxes[(samp * *nexon) + exon];
??? ??? ??? }
??? ??? }
??? ??? x_L[(6 * *nstate) + samp] = curmax / 0.999;
??? ??? x_U[(6 * *nstate) + samp] = 1000 * curmax;
??? }
??? 
??? for (int state = 0; state < *nstate; ++state) { 
??? ??? curmax = 0;
??? ??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? ??? if (statecols[samp] == state) { 
??? ??? ??? ??? curmax += x_L[(6 * *nstate) + samp];
??? ??? ??? }
??? ??? }
??? ??? curmax /= sampsperstate[state];
??? ??? x_L[(4 * *nstate) + state] = 0.0001 * curmax;
??? ??? x_U[(4 * *nstate) + state] = 10000 * curmax;
??? ??? x_L[(5 * *nstate) + state] = 0.0000001 * curmax;
??? ??? x_U[(5 * *nstate) + state] = 10000 * curmax;
??? }
??? 
??? for (int cons = 0; cons < *ncon; ++cons) { 
??? ??? g_L[cons] = 0.001;
??? ??? g_U[cons] = 2.0 * pow(10, 19);
??? }
??? ??? 
??? return true;
??? 
}


bool init_start_X()

{
??? 
??? for (int entry = 0; entry < 4 * *nstate; ++entry) {
??? ??? start_X[entry] = 1.0;
??? }
??? 
??? double readmax;
??? double statemeans[*nstate];
??? 
??? for (int state = 0; state < *nstate; ++state) { 
??? ??? statemeans[state] = 0;
??? }
??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? readmax = 0;
??? ??? for (int win = 0; win < *nwin; ++win) { 
??? ??? ???? if (reads[(samp * *nwin) + win] > readmax) { 
??? ??? ???? ??? readmax = reads[(samp * *nwin) + win];
??? ??? ???? }
??? ??? }
??? ??? start_X[(6 * *nstate) + samp] = 2.0 * readmax;
??? ??? statemeans[statecols[samp]] += 2.0 * readmax;
??? }
??? 
??? for (int state = 0; state < *nstate; ++state) { 
??? ??? start_X[(4 * *nstate) + state] = statemeans[state] / sampsperstate[state];
??? ??? start_X[(5 * *nstate) + state] = 0.1 * statemeans[state] / sampsperstate[state];
??? }
??? 
??? for (int entry = (6 * *nstate) + *nsamp; entry < *nvar; ++entry) { 
??? ??? start_X[entry] = 0.99;??? 
??? }
??? 
??? for (int entry = 0; entry < *nvar; ++entry) { 
??? ??? current_X[entry] = start_X[entry];
??? }
??? 
??? for (int entry = 0; entry < 7; ++entry) { 
??? ??? eval_counts[entry] = 0;
??? }
??? ??? ???? 
??? return true;??? ???? ??? ?

}


/* Likelihood and constraint calculations */

bool eval_ll(double *X_in, double &eval)

{

??? boost::shared_array<double> X(X_in);
??? evaluations::ll_full(*nsamp,*nwin,*nvar,statecols,
??? ??? exonrows,segrows,*nstate,*nexon,*nseg,
??? ??? reads,X,eval);
??? 
??? for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] = X[entry] ; }
??? *current_ll = eval;
??? eval_counts[0] += 1;
??? 
??? return true;

}


bool eval_cons(double *X, double *cons)

{ 

??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? cons[(samp * *nexon) + exon] = (X[(6 * *nstate) + samp] * X[(6 * *nstate) + *nsamp + (samp * *nexon) + exon]) - readmaxes[(samp * *nexon) + exon];
??? ??? }
??? }
??? 
??? for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] = X[entry] ; }
??? for (int entry = 0; entry < *ncon; ++entry) { current_cons[entry] = cons[entry] ; }
??? eval_counts[1] += 1;
??? ??? 
??? return true;
??? 
??? 
}


bool eval_ll_grad(double *X_in, double *grad_in)

{
??? boost::shared_array<double> X(X_in);
??? boost::shared_array<double> grad(grad_in);
??? evaluations::ll_grad_full(*nsamp,*nwin,*nvar,statecols,
??? ??? exonrows,segrows,*nstate,*nexon,*nseg,
??? ??? reads,X,grad);
??? 
??? for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] = X[entry] ; }
??? for (int entry = 0; entry < *nvar; ++entry) { current_ll_grad[entry] = grad[entry] ; }
??? eval_counts[2] += 1;
??? 
??? return true;

}


bool eval_cons_jac(double *X, double *jac)

{ 

??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? jac[2 * ((samp * *nexon) + exon)] = X[(6 * *nstate) + *nsamp + (samp * *nexon) + exon];
??? ??? ??? jac[(2 * ((samp * *nexon) + exon)) + 1] = X[(6 * *nstate) + samp];
??? ??? }
??? }
??? 
??? for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] = X[entry] ; }
??? for (int entry = 0; entry < *jac_entries; ++entry) { current_cons_jac[entry] = jac[entry] ; }
??? eval_counts[3] += 1;
??? 
??? return true;
??? 
}


bool eval_ll_hess(double *X_in, double *hess_in)

{
??? boost::shared_array<double> X(X_in);
??? boost::shared_array<double> hess(hess_in);
??? evaluations::ll_hess_full(*nsamp,*nwin,*nvar,*hess_entries,statecols,
??? ??? exonrows,segrows,*nstate,*nexon,*nseg,
??? ??? sampsperstate,segsperexon,reads,X,hess);
??? ??? 
??? for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] = X[entry] ; }
??? for (int entry = 0; entry < *hess_entries; ++entry) { current_ll_hess[entry] = hess[entry] ; }
??? eval_counts[4] += 1;
??? 
??? return true;

}


bool eval_cons_hess(double *X, double *hess)

{
??? for (int entry = 0; entry < *ncon; ++entry) { 
??? ??? hess[entry] = 1.0;
??? }
??? 
??? for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] = X[entry] ; }
??? for (int entry = 0; entry < *ncon; ++entry) { current_cons_hess[entry] = hess[entry] ; }
??? eval_counts[5] += 1;
??? 
??? return true;
??? 
}


bool eval_lagrangian(double *X, double &obj_factor, double *lambda, double *lagrangian)

{ 

??? int Nstart;

??? eval_ll_hess(X, lagrangian);
??? 
??? for (int entry = 0; entry < *hess_entries; ++entry) { 
??? ??? lagrangian[entry] *= obj_factor;
??? } 
??? 
??? for (int samp = 0; samp < *nsamp; ++samp) { 
??? ??? Nstart = (9 * *nstate) + (2 * *nseg * *nsamp) + (2 * *nexon * *nsamp) + (2 * *nsamp) + (samp * (1 + *nexon + *nseg));
??? ??? for (int exon = 0; exon < *nexon; ++exon) { 
??? ??? ??? lagrangian[Nstart + 1 + exon] += lambda[(samp * *nexon) + exon];
??? ??? }
??? }
??? 
??? for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] = X[entry] ; }
??? *current_obj_factor = obj_factor;
??? for (int entry = 0; entry < *ncon; ++entry) { current_lambda[entry] = lambda[entry] ; }
??? for (int entry = 0; entry < *hess_entries; ++entry) { current_lagrangian[entry] = lagrangian[entry] ; }
??? eval_counts[6] += 1;
??? 
??? return true;
??? 
}


bool eval_ll_py(numeric::array const &X_py, numeric::array const &eval_py) 

{

??? double *X = new double [*nvar];
??? double eval;
??? ??? 
??? for (int entry = 0; entry < *nvar; ++entry) { X[entry] = double(X_py[entry]) ; } 
??? 
??? eval_ll(X, eval);
??? 
??? eval_py[0] = eval;
??? 
??? return true;

}


bool eval_cons_py(numeric::array const &X_py, numeric::array const &cons_py)

{

??? double *X = new double [*nvar];
??? double *cons = new double [*ncon];
??? ??? ??? 
??? for (int entry = 0; entry < *nvar; ++entry) { X[entry] = double(X_py[entry]) ; } 
??? 
??? eval_cons(X, cons);
??? ??? 
??? for (int entry = 0; entry < *ncon; ++entry) { cons_py[entry] = cons[entry] ; } 
??? 
??? return true;

}


bool eval_ll_grad_py(numeric::array const &X_py, numeric::array const &grad_py)

{

??? double *X = new double [*nvar];
??? double *grad = new double [*nvar];
??? ??? ??? ??? 
??? for (int entry = 0; entry < *nvar; ++entry) { X[entry] = double(X_py[entry]) ; } 
??? 
??? eval_ll_grad(X, grad);
??? 
??? for (int entry = 0; entry < *nvar; ++entry) { grad_py[entry] = grad[entry] ; } 
??? 
??? return true;

}


bool eval_cons_jac_py(numeric::array const &X_py, numeric::array const &jac_py)

{

??? double *X = new double [*nvar];
??? double *jac = new double [*jac_entries];
??? ??? ??? ??? 
??? for (int entry = 0; entry < *nvar; ++entry) { X[entry] = double(X_py[entry]) ; } 
??? 
??? eval_cons_jac(X, jac);
??? ??? 
??? for (int entry = 0; entry < *jac_entries; ++entry) { jac_py[entry] = jac[entry] ; } 
??? 
??? return true;

}


bool eval_ll_hess_py(numeric::array const &X_py, numeric::array const &hess_py)

{

??? double *X = new double [*nvar];
??? double *hess = new double [*hess_entries];
??? ??? ??? ??? 
??? for (int entry = 0; entry < *nvar; ++entry) { X[entry] = double(X_py[entry]) ; } 
??? 
??? eval_ll_hess(X, hess);
??? ??? 
??? for (int entry = 0; entry < *hess_entries; ++entry) { hess_py[entry] = hess[entry] ; } 
??? 
??? return true;

}


bool eval_cons_hess_py(numeric::array const &X_py, numeric::array const &hess_py)

{

??? double *X = new double [*nvar];
??? double *hess = new double [*ncon];
??? ??? ??? ??? ??? 
??? for (int entry = 0; entry < *nvar; ++entry) { X[entry] = double(X_py[entry]) ; } 
??? 
??? eval_cons_hess(X, hess);
??? ??? 
??? for (int entry = 0; entry < *ncon; ++entry) { hess_py[entry] = hess[entry] ; } 
??? 
??? return true;

}


bool eval_lagrangian_py(numeric::array const &X_py, numeric::array const &obj_factor_py, numeric::array const &lambda_py, numeric::array const &lagrangian_py)

{

??? double *X = new double [*nvar];
??? double obj_factor;
??? double *lambda = new double [*ncon];
??? double *lagrangian = new double [*hess_entries];
??? ??? ??? ??? ??? 
??? for (int entry = 0; entry < *nvar; ++entry) { X[entry] = double(X_py[entry]) ; } 
??? 
??? obj_factor = double(obj_factor_py[0]);
??? ??? 
??? for (int entry = 0; entry < *ncon; ++entry) { lambda[entry] = double(lambda_py[entry]) ; } 
??? 
??? eval_lagrangian(X, obj_factor, lambda, lagrangian);
??? ??? 
??? for (int entry = 0; entry < *hess_entries; ++entry) { lagrangian_py[entry] = lagrangian[entry] ; } 
??? 
??? return true;

}

/* Variable access */

template <class T> bool update_var(std::string var_name, T var_value)

{

if (var_name == "gene_count") { gene_count = var_value ; }
if (var_name == "nstate") { *nstate = var_value ; }
if (var_name == "nsamp") { *nsamp = var_value ; }
if (var_name == "nexon") { *nexon = var_value ; }
if (var_name == "nseg") { *nseg = var_value ; }
if (var_name == "nwin") { *nwin = var_value ; }
if (var_name == "nvar") { *nvar = var_value ; }
if (var_name == "ncon") { *ncon = var_value ; }
if (var_name == "nsingle") { *nsingle = var_value ; }
if (var_name == "hsingle") { *hsingle = var_value ; }
if (var_name == "jac_entries") { *jac_entries = var_value ; }
if (var_name == "hess_entries") { *hess_entries = var_value ; }
if (var_name == "current_ll") { *current_ll = var_value ; }
if (var_name == "current_obj_factor") { *current_obj_factor = var_value ; }

}

template <class T> bool return_var(std::string var_name, T &var_value)

{

if (var_name == "gene_count") {var_value = gene_count ; }
if (var_name == "nstate") { var_value = *nstate ; }
if (var_name == "nsamp") { var_value = *nsamp ; }
if (var_name == "nexon") { var_value = *nexon ; }
if (var_name == "nseg") { var_value = *nseg ; }
if (var_name == "nwin") { var_value = *nwin ; }
if (var_name == "nvar") { var_value = *nvar ; }
if (var_name == "ncon") { var_value = *ncon ; }
if (var_name == "nsingle") { var_value = *nsingle ; }
if (var_name == "hsingle") { var_value = *hsingle ; }
if (var_name == "jac_entries") { var_value = *jac_entries ; }
if (var_name == "hess_entries") { var_value = *hess_entries ; }
if (var_name == "current_ll") { var_value = *current_ll ; }
if (var_name == "current_obj_factor") { var_value = *current_obj_factor ; }


}

template <class T> bool update_vec_var(std::string var_name, int var_length, T var_value[])

{

??? if (var_name == "statecols") {for (int entry = 0; entry < var_length; ++entry) { statecols[entry] = var_value[entry] ; } }
??? if (var_name == "sampcols") {for (int entry = 0; entry < var_length; ++entry) { sampcols[entry] = var_value[entry] ; } }
??? if (var_name == "exonrows") {for (int entry = 0; entry < var_length; ++entry) { exonrows[entry] = var_value[entry] ; } }
??? if (var_name == "segrows") {for (int entry = 0; entry < var_length; ++entry) { segrows[entry] = var_value[entry] ; } }
??? if (var_name == "sampsperstate") {for (int entry = 0; entry < var_length; ++entry) { sampsperstate[entry] = var_value[entry] ; } }
??? if (var_name == "segsperexon") {for (int entry = 0; entry < var_length; ++entry) { segsperexon[entry] = var_value[entry] ; } }
??? if (var_name == "single_hess_rows") {for (int entry = 0; entry < var_length; ++entry) { single_hess_rows[entry] = var_value[entry] ; } }
??? if (var_name == "single_hess_cols") {for (int entry = 0; entry < var_length; ++entry) { single_hess_cols[entry] = var_value[entry] ; } }
??? if (var_name == "jac_rows") {for (int entry = 0; entry < var_length; ++entry) { jac_rows[entry] = var_value[entry] ; } }
??? if (var_name == "jac_cols") {for (int entry = 0; entry < var_length; ++entry) { jac_cols[entry] = var_value[entry] ; } }
??? if (var_name == "full_hess_rows") {for (int entry = 0; entry < var_length; ++entry) { full_hess_rows[entry] = var_value[entry] ; } }
??? if (var_name == "full_hess_cols") {for (int entry = 0; entry < var_length; ++entry) { full_hess_cols[entry] = var_value[entry] ; } }
??? if (var_name == "reads") {for (int entry = 0; entry < var_length; ++entry) { reads[entry] = var_value[entry] ; } }
??? if (var_name == "readmaxes") {for (int entry = 0; entry < var_length; ++entry) { readmaxes[entry] = var_value[entry] ; } }
??? if (var_name == "eval_counts") {for (int entry = 0; entry < var_length; ++entry) { eval_counts[entry] = var_value[entry] ; } }
??? if (var_name == "start_X") {for (int entry = 0; entry < var_length; ++entry) { start_X[entry] = var_value[entry] ; } }
??? if (var_name == "x_L") {for (int entry = 0; entry < var_length; ++entry) { x_L[entry] = var_value[entry] ; } }
??? if (var_name == "x_U") {for (int entry = 0; entry < var_length; ++entry) { x_U[entry] = var_value[entry] ; } }
??? if (var_name == "g_L") {for (int entry = 0; entry < var_length; ++entry) { g_L[entry] = var_value[entry] ; } }
??? if (var_name == "g_U") {for (int entry = 0; entry < var_length; ++entry) { g_U[entry] = var_value[entry] ; } }
??? if (var_name == "current_X") {for (int entry = 0; entry < var_length; ++entry) { current_X[entry] = var_value[entry] ; } }
??? if (var_name == "current_cons") {for (int entry = 0; entry < var_length; ++entry) { current_cons[entry] = var_value[entry] ; } }
??? if (var_name == "current_ll_grad") {for (int entry = 0; entry < var_length; ++entry) { current_ll_grad[entry] = var_value[entry] ; } }
??? if (var_name == "current_cons_jac") {for (int entry = 0; entry < var_length; ++entry) { current_cons_jac[entry] = var_value[entry] ; } }
??? if (var_name == "current_ll_hess") {for (int entry = 0; entry < var_length; ++entry) { current_ll_hess[entry] = var_value[entry] ; } }
??? if (var_name == "current_cons_hess") {for (int entry = 0; entry < var_length; ++entry) { current_cons_hess[entry] = var_value[entry] ; } }
??? if (var_name == "current_lambda") {for (int entry = 0; entry < var_length; ++entry) { current_lambda[entry] = var_value[entry] ; } }
??? if (var_name == "current_lagrangian") {for (int entry = 0; entry < var_length; ++entry) { current_lagrangian[entry] = var_value[entry] ; } }

}

template <class T> bool return_vec_var(std::string var_name, int var_length, T var_value[])

{

??? if (var_name == "statecols") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = statecols[entry] ; } }
??? if (var_name == "sampcols") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = sampcols[entry] ; } }
??? if (var_name == "exonrows") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = exonrows[entry] ; } }
??? if (var_name == "segrows") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = segrows[entry] ; } }
??? if (var_name == "sampsperstate") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = sampsperstate[entry] ; } }
??? if (var_name == "segsperexon") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = segsperexon[entry] ; } }
??? if (var_name == "single_hess_rows") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = single_hess_rows[entry] ; } }
??? if (var_name == "single_hess_cols") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = single_hess_cols[entry] ; } }
??? if (var_name == "jac_rows") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = jac_rows[entry] ; } }
??? if (var_name == "jac_cols") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = jac_cols[entry] ; } }
??? if (var_name == "full_hess_rows") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = full_hess_rows[entry] ; } }
??? if (var_name == "full_hess_cols") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = full_hess_cols[entry] ; } }
??? if (var_name == "reads") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = reads[entry] ; } }
??? if (var_name == "readmaxes") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = readmaxes[entry] ; } }
??? if (var_name == "eval_counts") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = eval_counts[entry] ; } }
??? if (var_name == "start_X") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = start_X[entry] ; } }
??? if (var_name == "x_L") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = x_L[entry] ; } }
??? if (var_name == "x_U") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = x_U[entry] ; } }
??? if (var_name == "g_L") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = g_L[entry] ; } }
??? if (var_name == "g_U") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = g_U[entry] ; } }
??? if (var_name == "current_X") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = current_X[entry] ; } }
??? if (var_name == "current_cons") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = current_cons[entry] ; } }
??? if (var_name == "current_ll_grad") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = current_ll_grad[entry] ; } }
??? if (var_name == "current_cons_jac") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = current_cons_jac[entry] ; } }
??? if (var_name == "current_ll_hess") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = current_ll_hess[entry] ; } }
??? if (var_name == "current_cons_hess") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = current_cons_hess[entry] ; } }
??? if (var_name == "current_lambda") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = current_lambda[entry] ; } }
??? if (var_name == "current_lagrangian") {for (int entry = 0; entry < var_length; ++entry) { var_value[entry] = current_lagrangian[entry] ; } }

}

bool update_var_py(std::string var_name_py, numeric::array const &const &var_value_py, int var_length_py, std::string var_type_py)

{

??? if (var_type_py == "int") {
??? ??? if (var_length_py == 1) {
??? ??? ??? int var_value = int(var_value_py[0]);
??? ??? ??? update_var(var_name_py, var_value);
??? ??? } else {
??? ??? ??? int *var_value = new int [var_length_py];
??? ??? ??? for (int entry = 0; entry < var_length_py; ++entry) { 
??? ??? ??? ??? var_value[entry] = int(var_value_py[entry]);
??? ??? ??? }
??? ??? ??? update_vec_var(var_name_py, var_length_py, var_value);
??? ??? }
??? } else {
??? ??? if (var_length_py == 1) {
??? ??? ??? double var_value = double(var_value_py[0]);
??? ??? ??? update_var(var_name_py, var_value);
??? ??? } else {
??? ??? ??? double *var_value = new double [var_length_py];
??? ??? ??? for (int entry = 0; entry < var_length_py; ++entry) { 
??? ??? ??? ??? var_value[entry] = double(var_value_py[entry]);
??? ??? ??? }
??? ??? ??? update_vec_var(var_name_py, var_length_py, var_value);
??? ??? }
??? }

??? return true;
??? 
}


numeric::array const &return_var_py(std::string var_name_py, int var_length_py, std::string var_type_py)

{
??? list var_value_list_py = list();
??? 
??? if (var_type_py == "int") {
??? ??? if (var_length_py == 1) {
??? ??? ??? int var_value;
??? ??? ??? return_var(var_name_py, var_value);
??? ??? ??? var_value_py.append(int(var_value));
??? ??? } else {
??? ??? ??? int *var_value = new int [var_length_py];
??? ??? ??? return_vec_var(var_name_py, var_length_py, var_value);
??? ??? ??? for (int entry = 0; entry < var_length_py; ++entry) { 
??? ??? ??? ??? var_value_py.append(int(var_value[entry]));
??? ??? ??? }
??? ??? }
??? } else {
??? ??? if (var_length_py == 1) {
??? ??? ??? double var_value;
??? ??? ??? return_var(var_name_py, var_value);
??? ??? ??? var_value_py.append(double(var_value));
??? ??? } else {
??? ??? ??? double *var_value = new double [var_length_py];
??? ??? ??? return_vec_var(var_name_py, var_length_py, var_value);
??? ??? ??? for (int entry = 0; entry < var_length_py; ++entry) { 
??? ??? ??? ??? var_value_py.append(double(var_value[entry]));
??? ??? ??? }
??? ??? }
??? }
??? 
??? numeric::array const &var_value_array_py = numeric::array(var_value_list_py);

??? return var_value_array_py;
??? 
}


private:

int gene_count;

boost::shared_ptr<int> nstate;
boost::shared_ptr<int> nsamp;
boost::shared_ptr<int> nexon;
boost::shared_ptr<int> nseg;
boost::shared_ptr<int> nwin;
boost::shared_ptr<int> nvar;
boost::shared_ptr<int> ncon;
boost::shared_ptr<int> nsingle;
boost::shared_ptr<int> hsingle;
boost::shared_ptr<int> jac_entries;
boost::shared_ptr<int> hess_entries;

boost::shared_ptr<double> current_ll;
boost::shared_ptr<double> current_obj_factor;

boost::shared_array<int> statecols;
boost::shared_array<int> sampcols;
boost::shared_array<int> exonrows;
boost::shared_array<int> segrows;
boost::shared_array<int> sampsperstate;
boost::shared_array<int> segsperexon;
boost::shared_array<int> single_hess_rows;
boost::shared_array<int> single_hess_cols;
boost::shared_array<int> jac_rows;
boost::shared_array<int> jac_cols;
boost::shared_array<int> full_hess_rows;
boost::shared_array<int> full_hess_cols;
boost::shared_array<int> eval_counts;

boost::shared_array<double> reads;
boost::shared_array<double> readmaxes;
boost::shared_array<double> start_X;
boost::shared_array<double> x_L;
boost::shared_array<double> x_U;
boost::shared_array<double> g_L;
boost::shared_array<double> g_U;
boost::shared_array<double> current_X;
boost::shared_array<double> current_cons;
boost::shared_array<double> current_ll_grad;
boost::shared_array<double> current_cons_jac;
boost::shared_array<double> current_ll_hess;
boost::shared_array<double> current_cons_hess;
boost::shared_array<double> current_lambda;
boost::shared_array<double> current_lagrangian;
??? ??? ??? 
??? ??? ??? 
};


}


}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120512/772d63e4/attachment-0001.html>

From darkant at gmail.com  Sat May 12 22:28:28 2012
From: darkant at gmail.com (DarkAnt)
Date: Sat, 12 May 2012 16:28:28 -0400
Subject: [C++-sig] Nested Boost::Python dictionaries
In-Reply-To: <4FAEC220.4030005@gmail.com>
References: <CAOvfjcxXd7tA7SrChOaGXjZwOunQ0=RPd_9Wn2dAE0jEncZbew@mail.gmail.com>
	<4FAEC220.4030005@gmail.com>
Message-ID: <CAOvfjczt7A9ZAfKrM5ELeaX78FRHYa_kGJU-kd_e_qcSELv0eQ@mail.gmail.com>

Thanks Jim!

On Sat, May 12, 2012 at 4:03 PM, Jim Bosch <talljimbo at gmail.com> wrote:
> On 05/12/2012 12:22 PM, DarkAnt wrote:
>>
>> I'm trying to create a boost::python::dict that stores another
>> boost::python::dict.
>>
>> int main()
>> {
>> ? ? ? ?Py_Initialize();
>> ? ? ? ?boost::python::dict parent;
>> ? ? ? ?try{
>> ? ? ? ? ? ? ? ?parent["child_dict"] =
>> boost::make_shared<boost::python::dict>(boost::python::dict());
>> ? ? ? ?}
>> ? ? ? ?catch(...){
>> ? ? ? ? ? ? ? ?PyErr_Print();
>> ? ? ? ?}
>> ? ? ? ?return 0;
>> }
>>
>> TypeError: No to_python (by-value) converter found for C++ type: class
>> boost::shared_ptr<class boost::python::dict>
>>
>> I was under the impression that boost::shared_ptr had special
>> treatment in boost::python(that is the library already knew what to do
>> with it). I'm not quite sure if I'm supposed to write this to_python
>> converter or if I'm supposed to achieve this in a different manner. If
>> I do write the to_python converter what's the method of keeping track
>> of both reference counts?
>
>
> There's no need to use shared_ptr on Python objects; those will be tracked
> using Python's own reference counting system.
>
> Just doing
>
> parent["child_dict"] = boost::python::dict();
>
> will do what you want.
>
> Note that boost::python::dict (as well as boost::python::object, etc.) is
> actually a smart pointer itself, holding a PyObject* and
> incrementing/decrementing the reference count in the C++ copy constructor,
> assignment operator, and destructor.
>
> HTH!
>
> Jim
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig

From talljimbo at gmail.com  Sun May 13 00:41:50 2012
From: talljimbo at gmail.com (Jim Bosch)
Date: Sat, 12 May 2012 18:41:50 -0400
Subject: [C++-sig] Extensive memory errors upon C++ destruction and data
 transfer from numpy
In-Reply-To: <1336853437.41351.YahooMailNeo@web160302.mail.bf1.yahoo.com>
References: <1336853437.41351.YahooMailNeo@web160302.mail.bf1.yahoo.com>
Message-ID: <4FAEE72E.5010204@gmail.com>

Well, I'm afraid you're jumping into a number of thorny bushes at once 
here, trying to make C++ code talk to NumPy through Boost.Python while 
learning C++.  So I'll start with a few principles:

1) Never call a destructor yourself, in C++ or in Python.  That should 
always happen automatically when an object goes out of scope unless 
you're doing really advanced things (and you'll know when that is when 
you learn about those advanced things).  Doing this is probably the 
source of essentially all your problems.  However, "del obj" isn't 
really a direct call to a destructor in Python; it just removes one 
reference to a variable, which might call the destructor if that's the 
last reference Python knows about - so you can do that if you like, but 
it's usually unnecessary because the same thing happens automatically 
when a variable goes out of scope or is reassigned.

2) shared_ptr is great, but I'm worried you might be using it more than 
you need to, especially enable_shared_from_this.  Sometimes 
enable_shared_from_this is necessary, but often it's a sign you could 
have designed something better.  Because Python does its own reference 
counting, you'll have an easier time figuring out what's going on with 
your memory if you only have one kind of reference counter floating 
around your code.

3) You seem to be using pointers as arguments for C++ functions more 
often that I'd consider necessary.  And pointer arguments to numeric 
types will not work well with Python, because numbers are immutable in 
Python.  For instance, all those ints in register_new_gene should 
probably be passed by value if you want them to work sensibly with Python.

4) numeric_array is a little old and limited, but those limitations may 
have saved you from creating even bigger memory headaches for you, 
because as far as I can tell there's no way to create NumPy arrays to 
C++ memory from it, or extract C++ arrays from a numeric::array object. 
  That means you have to copy everything, which is inefficient, but it 
also means you don't have to worry about aliasing and ownership of array 
memory, which I think probably the right choice for right now.

Finally, I'd really recommend making this work in plain C++, even in a 
limited fashion, before throwing it into Python.  The Python/C++ barrier 
is tricky even with the best tools, and as a C++ beginner you'll have a 
much easier time tracking down memory issues using plain C++ and a 
debugger and/or a tool like valgrind.

I haven't looked closely at your code yet, but I'm hoping the above will 
give you enough to work on that it will be substantially different 
before I need to anyhow :)

HTH!

Jim




On 05/12/2012 04:10 PM, Christopher Cowing-Zitron wrote:
> Hello,
>
> I am writing a computational package integrating Python and C++ using
> Boost.Python. The interface and the initial data collection are in
> Python, but the workhorse data structures and computation are in C++. I
> am having difficulties with memory errors at two points: at the transfer
> of data between numpy and C++, and upon the destruction of wrapped C++
> objects. I suspect the issues are related, and are not the result of
> limitations of Python, C++, or Boost.Python, but rather are the result
> of my lack of familiarity with C++. I apologize for the long email, but
> my lack of experience with C++ makes it hard to describe my problems
> succinctly, especially because I have no idea where the problems are
> occurring, nor do I know the right vocabulary. I've pasted below the C++
> source for the relevant class, as well as the Boost.Python module
> wrapper code. This is all for academic research, and under the legal
> conditions of my grants it is necessarily open-source, but because I
> intend to publish this package in a research journal, I'd appreciate it
> if you didn't share the code beyond this mailing list; I'll create pypi
> and sourceforge entries for the source once it's ready for release. Most
> importantly, feel free to talk trash about and point out any errors in
> my code: I just started with C++ three weeks ago, and criticism is very
> helpful and much appreciated! :-)
>
> My package is for the analysis of large biological data sets, and a new
> C++ object must be created for each gene analyzed, which for the human
> genome amounts to about 30,000 times for the analysis of a single data
> set. A single gene object stores many variables, some of which are
> scalar and could be built into the class as static variables, but most
> variables are array data which greatly vary in size (and possibly
> dimension) depending upon the biological structure of the individual
> gene as well as the nature of the experiment being analyze, which cannot
> be known in advance. As such, all the array variables are dynamically
> allocated. I have used Boost.Python shared pointers and shared arrays
> for all dynamically allocated variables. In an attempt at uniformity, in
> this version of the code, all gene class variables are dynamically
> allocated.
>
> I have separated the creation of a gene object and the filling of
> variable values into two steps. All access of C++ variables is done
> through wrapped get/set methods, and all data is transferred via numpy
> arrays. I have no problems upon object creation, but I get memory errors
> both when filling the variables from numpy arrays, and transferring
> calculated variable values back into numpy arrays. The issue seems to be
> that I cannot force a copy of variable values between numpy arrays and
> C++, but rather Boost.Python seems intent upon using references (in fact
> the documentation for the extract<> template explicitly states that, and
> I can't find any other way to get variables out of numpy arrays and into
> C++). This matters because the C++ gene object will modify some of its
> own variable values that were initially taken from Python, and the user
> will similarly modify within Python variables used to fill the gene
> object, and I need to make sure changes in one do not get automatically
> reflected in the other. In both cases I get bus errors / segmentation
> faults.
>
> The second problem relates to object deletion. Once a single gene has
> been analyzed, neither the C++ object nor the corresponding Python
> object are needed; the important numerical results are simply stored in
> a numpy array. Because of the huge number of genes, I have one of two
> options when it comes to memory management, in order to avoid the
> build-up of excess memory consumption by genes for which the analysis
> has been completed: I can either explicitly destroy the C++ object, by
> brute force or by reassigning a new object to its variable in Python, or
> I can reinitialize its variables to store a new gene (I am inclined to
> the second). But whenever I try to destroy a wrapped C++ object, using
> either method, I get a segmentation fault. I've written a C++ destructor
> for my gene class; I tried to wrap it for Python, but the Boost.Python
> documentation doesn't mention how to register destructors to Python, so
> I just wrapped it like any other bound method. (Note: that wrapper is
> not present in the version I've posted below, because I assumed I got it
> wrong anyway). I still get the faults using that method, as called by my
> assigned name. Should I be using the Python object's .__del__ method or
> the global Python del(), and if so, how do I specialize them to my gene
> object's destructor? Even if I go with the option of just reinitializing
> the C++ gene object variables, rather than explicit deletion, I still
> get a segmentation fault when I leave the Python interpreter, presumably
> because it's trying to destroy the wrapped C++ gene objects created in
> that session.
>
> Any and all help and advice is very much appreciated, and I thank you in
> advance for your help. If any of you provide particular help, I will be
> happy to include you in the acknowledgements should I get this darn
> thing published; I'd offer co-authorship, but my boss would never allow
> it. Have a good weekend.
>
> -- Chris Cowing-Zitron
>
> fp_boosters.hpp, the source file defining the Boost.Python module:
>
> #include <fp_genes.hpp>
>
> BOOST_PYTHON_MODULE(fp_boosters)
> {
> def("loggamma", &fingerpuppet::evaluations::loggamma_py,
> return_value_policy<return_by_value>());
> def("digamma", &fingerpuppet::evaluations::digamma_py,
> return_value_policy<return_by_value>());
> def("trigamma", &fingerpuppet::evaluations::trigamma_py,
> return_value_policy<return_by_value>());
> def("ll_single", &fingerpuppet::evaluations::ll_single_py);
> def("ll_grad_single", &fingerpuppet::evaluations::ll_grad_single_py);
> def("ll_hess_single", &fingerpuppet::evaluations::ll_hess_single_py);
> def("ll_full", &fingerpuppet::evaluations::ll_full_py);
> def("ll_grad_full", &fingerpuppet::evaluations::ll_grad_full_py);
> def("ll_hess_full", &fingerpuppet::evaluations::ll_hess_full_py);
>
> class_<fingerpuppet::genes::gene, boost::noncopyable>("gene")
>
> .def("register_gene", &fingerpuppet::genes::gene::register_new_gene_py)
> .def("reset_gene", &fingerpuppet::genes::gene::reset_gene)
> .def("empty_gene", &fingerpuppet::genes::gene::empty_gene)
> .def("eval_ll", &fingerpuppet::genes::gene::eval_ll_py)
> .def("eval_cons", &fingerpuppet::genes::gene::eval_cons_py)
> .def("eval_ll_grad", &fingerpuppet::genes::gene::eval_ll_grad_py)
> .def("eval_cons_jac", &fingerpuppet::genes::gene::eval_cons_jac_py)
> .def("eval_ll_hess", &fingerpuppet::genes::gene::eval_ll_hess_py)
> .def("eval_cons_hess", &fingerpuppet::genes::gene::eval_cons_hess_py)
> .def("eval_lagrangian", &fingerpuppet::genes::gene::eval_lagrangian_py)
> .def("update_var", &fingerpuppet::genes::gene::update_var_py)
> .def("return_var", &fingerpuppet::genes::gene::return_var_py)
>
> ;
>
> numeric::array::set_module_and_type("numpy", "ndarray");
> };
>
>
>
> fp_genes.hpp, the header defining the gene object class:
>
> #include <fp_evaluations.hpp>
>
>
> namespace fingerpuppet {
>
>
> namespace genes {
>
> class gene;
>
> class gene : public boost::enable_shared_from_this<gene>, public
> boost::noncopyable
>
> {
>
>
> public:
>
>
> /* Constructors, destructor, and construction utilities */
>
> gene()
>
>
> {
> gene_count = 0;
> reset_gene();
>
> }
>
>
> ~gene()
>
> {
>
> empty_gene();
>
> }
>
>
> boost::shared_ptr<gene> return_this()
>
> {
>
> return shared_from_this();
>
> }
>
>
> bool register_new_gene(int *ns, int *statecolsin, int *sampcolsin, int
> *exonrowsin, int *segrowsin, double *readsin)
>
> {
>
> reset_gene();
> gene_count += 1;
>
> nstate.reset(new int);
> nsamp.reset(new int);
> nexon.reset(new int);
> nseg.reset(new int);
> nwin.reset(new int);
> nvar.reset(new int);
> ncon.reset(new int);
> nsingle.reset(new int);
> hsingle.reset(new int);
> jac_entries.reset(new int);
> hess_entries.reset(new int);
> current_ll.reset(new double);
> current_obj_factor.reset(new double);
>
> *nstate = ns[0];
> *nsamp = ns[1];
> *nexon = ns[2];
> *nseg = ns[3];
> *nwin = ns[4];
> *nvar = ns[5];
> *ncon = *nsamp * *nexon;
> *nsingle = 9;
> *hsingle = 21;
> *jac_entries = 2 * *ncon;
> *hess_entries = 0;
> *current_ll = 0.0;
> *current_obj_factor = 1.0;
>
> statecols.reset(new int [*nsamp]);
> sampcols.reset(new int [*nsamp]);
> exonrows.reset(new int [*nwin]);
> segrows.reset(new int [*nwin]);
> sampsperstate.reset(new int [*nstate]);
> segsperexon.reset(new int [*nseg]);
> single_hess_rows.reset(new int [*hsingle]);
> single_hess_cols.reset(new int [*hsingle]);
> jac_rows.reset(new int [*jac_entries]);
> jac_cols.reset(new int [*jac_entries]);
> eval_counts.reset(new int [7]);
>
> reads.reset(new double [*nsamp * *nwin]);
> readmaxes.reset(new double [*nsamp * *nexon]);
> start_X.reset(new double [*nvar]);
> x_L.reset(new double [*nvar]);
> x_U.reset(new double [*nvar]);
> g_L.reset(new double [*ncon]);
> g_U.reset(new double [*ncon]);
> current_X.reset(new double [*nvar]);
> current_cons.reset(new double [*ncon]);
> current_ll_grad.reset(new double [*nvar]);
> current_cons_jac.reset(new double [*jac_entries]);
> current_cons_hess.reset(new double [*ncon]);
> current_lambda.reset(new double [*ncon]);
>
>
> for (int entry = 0; entry < *ncon; ++entry) {
> readmaxes[entry] = 0.0;
> }
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int win = 0; win < *nwin; ++win) {
> if (reads[(samp * *nwin) + win] > readmaxes[(samp * *nexon) +
> exonrows[win]]) {
> readmaxes[(samp * *nexon) + exonrows[win]] = reads[(samp * *nwin) + win];
> }
> }
> }
>
> for (int entry = 0; entry < *nstate; ++entry) {
> sampsperstate[entry] = 0;
> }
>
> for (int entry = 0; entry < *nsamp; ++entry) {
> sampsperstate[statecols[entry]] += 1;
> }
>
> for (int entry = 0; entry < *nexon; ++entry) {
> segsperexon[entry] = 0;
> }
>
> segsperexon[0] = 1;
>
> for (int entry = 1; entry < *nwin; ++entry) {
> if (segrows[entry] != segrows[entry - 1]) {
> segsperexon[exonrows[entry]] += 1;
> }
> }
>
> init_jac_struct();
> init_hess_struct();
> init_bounds();
> init_start_X();
>
>
> }
>
>
> bool register_new_gene_py(numeric::array const &ns_py, numeric::array
> const &statecolsin_py, numeric::array const &sampcolsin_py,
> numeric::array const &exonrowsin_py, numeric::array const &segrowsin_py,
> numeric::array const &readsin_py)
>
>
> {
>
> reset_gene();
> gene_count += 1;
>
> nstate.reset(new int);
> nsamp.reset(new int);
> nexon.reset(new int);
> nseg.reset(new int);
> nwin.reset(new int);
> nvar.reset(new int);
> ncon.reset(new int);
> nsingle.reset(new int);
> hsingle.reset(new int);
> jac_entries.reset(new int);
> hess_entries.reset(new int);
> current_ll.reset(new double);
> current_obj_factor.reset(new double);
>
> *nstate = int(ns_py[0]);
> *nsamp = int(ns_py[1]);
> *nexon = int(ns_py[2]);
> *nseg = int(ns_py[3]);
> *nwin = int(ns_py[4]);
> *nvar = int(ns_py[5]);
> *ncon = *nsamp * *nexon;
> *nsingle = 9;
> *hsingle = 21;
> *jac_entries = 2 * *ncon;
> *hess_entries = 0;
> *current_ll = 0.0;
> *current_obj_factor = 1.0;
>
> statecols.reset(new int [*nsamp]);
> sampcols.reset(new int [*nsamp]);
> exonrows.reset(new int [*nwin]);
> segrows.reset(new int [*nwin]);
> sampsperstate.reset(new int [*nstate]);
> segsperexon.reset(new int [*nseg]);
> single_hess_rows.reset(new int [*hsingle]);
> single_hess_cols.reset(new int [*hsingle]);
> jac_rows.reset(new int [*jac_entries]);
> jac_cols.reset(new int [*jac_entries]);
> eval_counts.reset(new int [7]);
>
> reads.reset(new double [*nsamp * *nwin]);
> readmaxes.reset(new double [*nsamp * *nexon]);
> start_X.reset(new double [*nvar]);
> x_L.reset(new double [*nvar]);
> x_U.reset(new double [*nvar]);
> g_L.reset(new double [*ncon]);
> g_U.reset(new double [*ncon]);
> current_X.reset(new double [*nvar]);
> current_cons.reset(new double [*ncon]);
> current_ll_grad.reset(new double [*nvar]);
> current_cons_jac.reset(new double [*jac_entries]);
> current_cons_hess.reset(new double [*ncon]);
> current_lambda.reset(new double [*ncon]);
>
> for (int entry = 0; entry < *nsamp; ++entry) {
> statecols[entry] = extract<int>(statecolsin_py[entry]());
> }
>
> for (int entry = 0; entry < *nsamp; ++entry) {
> sampcols[entry] = extract<int>(sampcolsin_py[entry]());
> }
>
> for (int entry = 0; entry < *nwin; ++entry) {
> exonrows[entry] = extract<int>(exonrowsin_py[entry]());
> }
>
> for (int entry = 0; entry < *nwin; ++entry) {
> segrows[entry] = extract<int>(segrowsin_py[entry]());
> }
>
> for (int entry = 0; entry < *nsamp * *nwin; ++entry) {
> reads[entry] = extract<double>(readsin_py[entry]());
> }
>
> for (int entry = 0; entry < *ncon; ++entry) {
> readmaxes[entry] = 0.0;
> }
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int win = 0; win < *nwin; ++win) {
> if (reads[(samp * *nwin) + win] > readmaxes[(samp * *nexon) +
> exonrows[win]]) {
> readmaxes[(samp * *nexon) + exonrows[win]] = reads[(samp * *nwin) + win];
> }
> }
> }
>
> for (int entry = 0; entry < *nstate; ++entry) {
> sampsperstate[entry] = 0;
> }
>
> for (int entry = 0; entry < *nsamp; ++entry) {
> sampsperstate[statecols[entry]] += 1;
> }
>
> for (int entry = 0; entry < *nexon; ++entry) {
> segsperexon[entry] = 0;
> }
>
> segsperexon[0] = 1;
>
> for (int entry = 1; entry < *nwin; ++entry) {
> if (segrows[entry] != segrows[entry - 1]) {
> segsperexon[exonrows[entry]] += 1;
> }
> }
>
> init_jac_struct();
> init_hess_struct();
> init_bounds();
> init_start_X();
>
> }
>
> bool reset_gene()
>
> {
>
> nstate.reset(new int);
> nsamp.reset(new int);
> nexon.reset(new int);
> nseg.reset(new int);
> nwin.reset(new int);
> nvar.reset(new int);
> ncon.reset(new int);
> nsingle.reset(new int);
> hsingle.reset(new int);
> jac_entries.reset(new int);
> hess_entries.reset(new int);
> current_ll.reset(new double);
> current_obj_factor.reset(new double);
> statecols.reset(new int [1]);
> sampcols.reset(new int [1]);
> exonrows.reset(new int [1]);
> segrows.reset(new int [1]);
> sampsperstate.reset(new int [1]);
> segsperexon.reset(new int [1]);
> single_hess_rows.reset(new int [1]);
> single_hess_cols.reset(new int [1]);
> jac_rows.reset(new int [1]);
> jac_cols.reset(new int [1]);
> full_hess_rows.reset(new int [1]);
> full_hess_cols.reset(new int [1]);
> eval_counts.reset(new int [1]);
> reads.reset(new double [1]);
> readmaxes.reset(new double [1]);
> start_X.reset(new double [1]);
> x_L.reset(new double [1]);
> x_U.reset(new double [1]);
> g_L.reset(new double [1]);
> g_U.reset(new double [1]);
> current_X.reset(new double [1]);
> current_cons.reset(new double [1]);
> current_ll_grad.reset(new double [1]);
> current_cons_jac.reset(new double [1]);
> current_ll_hess.reset(new double [1]);
> current_cons_hess.reset(new double [1]);
> current_lambda.reset(new double [1]);
> current_lagrangian.reset(new double [1]);
>
> return true;
>
> }
>
>
> bool empty_gene()
>
> {
>
> nstate.reset();
> nsamp.reset();
> nexon.reset();
> nseg.reset();
> nwin.reset();
> nvar.reset();
> ncon.reset();
> nsingle.reset();
> hsingle.reset();
> jac_entries.reset();
> hess_entries.reset();
> current_ll.reset();
> current_obj_factor.reset();
> statecols.reset();
> sampcols.reset();
> exonrows.reset();
> segrows.reset();
> sampsperstate.reset();
> segsperexon.reset();
> single_hess_rows.reset();
> single_hess_cols.reset();
> jac_rows.reset();
> jac_cols.reset();
> full_hess_rows.reset();
> full_hess_cols.reset();
> eval_counts.reset();
> reads.reset();
> readmaxes.reset();
> start_X.reset();
> x_L.reset();
> x_U.reset();
> g_L.reset();
> g_U.reset();
> current_X.reset();
> current_cons.reset();
> current_ll_grad.reset();
> current_cons_jac.reset();
> current_ll_hess.reset();
> current_cons_hess.reset();
> current_lambda.reset();
> current_lagrangian.reset();
> return true;
>
> }
>
>
> bool init_jac_struct()
>
> {
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int exon = 0; exon < *nexon; ++exon) {
> jac_rows[2 * ((samp * *nexon) + exon)] = (6 * *nstate) + samp;
> jac_cols[2 * ((samp * *nexon) + exon)] = (samp * *nexon) + exon;
> jac_rows[(2 * ((samp * *nexon) + exon)) + 1] = (6 * *nstate) + *nsamp +
> (samp * *nexon) + exon;
> jac_cols[(2 * ((samp * *nexon) + exon)) + 1] = (samp * *nexon) + exon;
> }
> }
>
> return true;
>
> }
>
>
> bool init_hess_struct()
>
> {
>
> int shr[21] = {3,4,2,4,2,5,6,1,6,1,7,8,0,8,0,0,1,2,1,2,2};
> int shc[21] = {3,3,3,4,4,5,5,5,6,6,7,7,7,8,8,0,0,0,1,1,2};
>
> for (int entry = 0; entry < 21; ++entry) {
> single_hess_rows[entry] = shr[entry];
> single_hess_cols[entry] = shc[entry];
> }
>
> for (int state = 0; state < *nstate; ++state) {
> *hess_entries += 3 + (2 * *nseg * sampsperstate[state]);
> }
>
> for (int state = 0; state < *nstate; ++state) {
> *hess_entries += 3 + (2 * *nexon * sampsperstate[state]);
> }
>
> for (int state = 0; state < *nstate; ++state) {
> *hess_entries += 3 + (2 * sampsperstate[state]);
> }
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> *hess_entries += 1 + *nexon + *nseg;
> }
>
> for (int exon = 0; exon < *nexon; ++exon) {
> *hess_entries += *nsamp * (1 + segsperexon[exon]);
> }
>
> *hess_entries += *nsamp * *nseg;
>
> full_hess_rows.reset(new int [*hess_entries]);
> full_hess_cols.reset(new int [*hess_entries]);
> current_ll_hess.reset(new double [*hess_entries]);
> current_lagrangian.reset(new double [*hess_entries]);
>
> int entry = 0;
>
> for (int state = 0; state < *nstate; ++state) {
> for (int var = 0; var < 2 + (*nseg * sampsperstate[state]); ++var) {
> full_hess_cols[entry + var] = state;
> }
> full_hess_rows[entry] = state;
> full_hess_rows[entry + 1] = *nstate + state;
> entry += 2;
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int seg = 0; seg < *nseg; ++seg) {
> if (statecols[samp] == state) {
> full_hess_rows[entry] = (6 * *nstate) + (*nsamp * (*nexon + 1)) + (samp
> * *nseg) + seg;
> entry += 1;
> }
> }
> }
> }
>
> for (int state = 0; state < *nstate; ++state) {
> for (int var = 0; var < 1 + (*nseg * sampsperstate[state]); ++var) {
> full_hess_cols[entry + var] = *nstate + state;
> }
> full_hess_rows[entry] = *nstate + state;
> entry += 1;
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int seg = 0; seg < *nseg; ++seg) {
> if (statecols[samp] == state) {
> full_hess_rows[entry] = (6 * *nstate) + (*nsamp * (*nexon + 1)) + (samp
> * *nseg) + seg;
> entry += 1;
> }
> }
> }
> }
>
> for (int state = 0; state < *nstate; ++state) {
> for (int var = 0; var < 2 + (*nexon * sampsperstate[state]); ++var) {
> full_hess_cols[entry + var] = (2 * *nstate) + state;
> }
> full_hess_rows[entry] = (2 * *nstate) + state;
> full_hess_rows[entry + 1] = (3 * *nstate) + state;
> entry += 2;
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int exon = 0; exon < *nexon; ++exon) {
> if (statecols[samp] == state) {
> full_hess_rows[entry] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
> entry += 1;
> }
> }
> }
> }
>
> for (int state = 0; state < *nstate; ++state) {
> for (int var = 0; var < 1 + (*nexon * sampsperstate[state]); ++var) {
> full_hess_cols[entry + var] = (3 * *nstate) + state;
> }
> full_hess_rows[entry] = (3 * *nstate) + state;
> entry += 1;
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int exon = 0; exon < *nexon; ++exon) {
> if (statecols[samp] == state) {
> full_hess_rows[entry] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
> entry += 1;
> }
> }
> }
> }
>
> for (int state = 0; state < *nstate; ++state) {
> for (int var = 0; var < 2 + sampsperstate[state]; ++var) {
> full_hess_cols[entry + var] = (4 * *nstate) + state;
> }
> full_hess_rows[entry] = (4 * *nstate) + state;
> full_hess_rows[entry + 1] = (5 * *nstate) + state;
> entry += 2;
> for (int samp = 0; samp < *nsamp; ++samp) {
> if (statecols[samp] == state) {
> full_hess_rows[entry] = (6 * *nstate) + samp;
> entry += 1;
> }
> }
> }
>
> for (int state = 0; state < *nstate; ++state) {
> for (int var = 0; var < 1 + sampsperstate[state]; ++var) {
> full_hess_cols[entry + var] = (5 * *nstate) + state;
> }
> full_hess_rows[entry] = (5 * *nstate) + state;
> entry += 1;
> for (int samp = 0; samp < *nsamp; ++samp) {
> if (statecols[samp] == state) {
> full_hess_rows[entry] = (6 * *nstate) + samp;
> entry += 1;
> }
> }
> }
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int var = 0; var < 1 + *nexon + *nseg; ++var) {
> full_hess_cols[entry + var] = (6 * *nstate) + samp;
> }
> full_hess_rows[entry] = (6 * *nstate) + samp;
> entry += 1;
> for (int exon = 0; exon < *nexon; ++exon) {
> full_hess_rows[entry] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
> entry += 1;
> }
> for (int seg = 0; seg < *nseg; ++seg) {
> full_hess_rows[entry] = (6 * *nstate) + (*nsamp * (*nexon + 1)) + (samp
> * *nseg) + seg;
> entry += 1;
> }
> }
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> int startseg = 0;
> for (int exon = 0; exon < *nexon; ++exon) {
> for (int var = 0; var < 1 + segsperexon[exon]; ++var) {
> full_hess_cols[entry + var] = (6 * *nstate) + *nsamp + (samp * *nexon) +
> exon;
> }
> startseg += segsperexon[exon];
> full_hess_rows[entry] = (6 * *nstate) + *nsamp + (samp * *nexon) + exon;
> entry += 1;
> for (int seg = startseg; seg < startseg + segsperexon[exon]; ++seg) {
> full_hess_rows[entry] = (6 * *nstate) + (*nsamp * (*nexon + 1)) + (samp
> * *nseg) + seg;
> entry += 1;
> }
> }
> }
>
> for (int var = (6 * *nstate) + (*nsamp * (*nexon + 1)); var < *nvar;
> ++var) {
> full_hess_cols[entry] = var;
> full_hess_rows[entry] = var;
> entry += 1;
> }
>
> return true;
>
> }
>
>
> bool init_bounds()
>
> {
>
> double curmax;
>
> for (int entry = 0; entry < 4 * *nvar; ++entry) {
> x_L[entry] = 0.01;
> x_U[entry] = 100.0;
> }
>
> for (int entry = (6 * *nstate) + *nsamp; entry < *nvar; ++entry) {
> x_L[entry] = 0.001;
> x_U[entry] = 0.999;
> }
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> curmax = 0;
> for (int exon = 0; exon < *nexon; ++exon) {
> if (readmaxes[(samp * *nexon) + exon] > curmax) {
> curmax = readmaxes[(samp * *nexon) + exon];
> }
> }
> x_L[(6 * *nstate) + samp] = curmax / 0.999;
> x_U[(6 * *nstate) + samp] = 1000 * curmax;
> }
>
> for (int state = 0; state < *nstate; ++state) {
> curmax = 0;
> for (int samp = 0; samp < *nsamp; ++samp) {
> if (statecols[samp] == state) {
> curmax += x_L[(6 * *nstate) + samp];
> }
> }
> curmax /= sampsperstate[state];
> x_L[(4 * *nstate) + state] = 0.0001 * curmax;
> x_U[(4 * *nstate) + state] = 10000 * curmax;
> x_L[(5 * *nstate) + state] = 0.0000001 * curmax;
> x_U[(5 * *nstate) + state] = 10000 * curmax;
> }
>
> for (int cons = 0; cons < *ncon; ++cons) {
> g_L[cons] = 0.001;
> g_U[cons] = 2.0 * pow(10, 19);
> }
>
> return true;
>
> }
>
>
> bool init_start_X()
>
> {
>
> for (int entry = 0; entry < 4 * *nstate; ++entry) {
> start_X[entry] = 1.0;
> }
>
> double readmax;
> double statemeans[*nstate];
>
> for (int state = 0; state < *nstate; ++state) {
> statemeans[state] = 0;
> }
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> readmax = 0;
> for (int win = 0; win < *nwin; ++win) {
> if (reads[(samp * *nwin) + win] > readmax) {
> readmax = reads[(samp * *nwin) + win];
> }
> }
> start_X[(6 * *nstate) + samp] = 2.0 * readmax;
> statemeans[statecols[samp]] += 2.0 * readmax;
> }
>
> for (int state = 0; state < *nstate; ++state) {
> start_X[(4 * *nstate) + state] = statemeans[state] / sampsperstate[state];
> start_X[(5 * *nstate) + state] = 0.1 * statemeans[state] /
> sampsperstate[state];
> }
>
> for (int entry = (6 * *nstate) + *nsamp; entry < *nvar; ++entry) {
> start_X[entry] = 0.99;
> }
>
> for (int entry = 0; entry < *nvar; ++entry) {
> current_X[entry] = start_X[entry];
> }
>
> for (int entry = 0; entry < 7; ++entry) {
> eval_counts[entry] = 0;
> }
>
> return true;
>
> }
>
>
> /* Likelihood and constraint calculations */
>
> bool eval_ll(double *X_in, double &eval)
>
> {
>
> boost::shared_array<double> X(X_in);
> evaluations::ll_full(*nsamp,*nwin,*nvar,statecols,
> exonrows,segrows,*nstate,*nexon,*nseg,
> reads,X,eval);
>
> for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] =
> X[entry] ; }
> *current_ll = eval;
> eval_counts[0] += 1;
>
> return true;
>
> }
>
>
> bool eval_cons(double *X, double *cons)
>
> {
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int exon = 0; exon < *nexon; ++exon) {
> cons[(samp * *nexon) + exon] = (X[(6 * *nstate) + samp] * X[(6 *
> *nstate) + *nsamp + (samp * *nexon) + exon]) - readmaxes[(samp * *nexon)
> + exon];
> }
> }
>
> for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] =
> X[entry] ; }
> for (int entry = 0; entry < *ncon; ++entry) { current_cons[entry] =
> cons[entry] ; }
> eval_counts[1] += 1;
>
> return true;
>
>
> }
>
>
> bool eval_ll_grad(double *X_in, double *grad_in)
>
> {
> boost::shared_array<double> X(X_in);
> boost::shared_array<double> grad(grad_in);
> evaluations::ll_grad_full(*nsamp,*nwin,*nvar,statecols,
> exonrows,segrows,*nstate,*nexon,*nseg,
> reads,X,grad);
>
> for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] =
> X[entry] ; }
> for (int entry = 0; entry < *nvar; ++entry) { current_ll_grad[entry] =
> grad[entry] ; }
> eval_counts[2] += 1;
>
> return true;
>
> }
>
>
> bool eval_cons_jac(double *X, double *jac)
>
> {
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> for (int exon = 0; exon < *nexon; ++exon) {
> jac[2 * ((samp * *nexon) + exon)] = X[(6 * *nstate) + *nsamp + (samp *
> *nexon) + exon];
> jac[(2 * ((samp * *nexon) + exon)) + 1] = X[(6 * *nstate) + samp];
> }
> }
>
> for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] =
> X[entry] ; }
> for (int entry = 0; entry < *jac_entries; ++entry) {
> current_cons_jac[entry] = jac[entry] ; }
> eval_counts[3] += 1;
>
> return true;
>
> }
>
>
> bool eval_ll_hess(double *X_in, double *hess_in)
>
> {
> boost::shared_array<double> X(X_in);
> boost::shared_array<double> hess(hess_in);
> evaluations::ll_hess_full(*nsamp,*nwin,*nvar,*hess_entries,statecols,
> exonrows,segrows,*nstate,*nexon,*nseg,
> sampsperstate,segsperexon,reads,X,hess);
>
> for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] =
> X[entry] ; }
> for (int entry = 0; entry < *hess_entries; ++entry) {
> current_ll_hess[entry] = hess[entry] ; }
> eval_counts[4] += 1;
>
> return true;
>
> }
>
>
> bool eval_cons_hess(double *X, double *hess)
>
> {
> for (int entry = 0; entry < *ncon; ++entry) {
> hess[entry] = 1.0;
> }
>
> for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] =
> X[entry] ; }
> for (int entry = 0; entry < *ncon; ++entry) { current_cons_hess[entry] =
> hess[entry] ; }
> eval_counts[5] += 1;
>
> return true;
>
> }
>
>
> bool eval_lagrangian(double *X, double &obj_factor, double *lambda,
> double *lagrangian)
>
> {
>
> int Nstart;
>
> eval_ll_hess(X, lagrangian);
>
> for (int entry = 0; entry < *hess_entries; ++entry) {
> lagrangian[entry] *= obj_factor;
> }
>
> for (int samp = 0; samp < *nsamp; ++samp) {
> Nstart = (9 * *nstate) + (2 * *nseg * *nsamp) + (2 * *nexon * *nsamp) +
> (2 * *nsamp) + (samp * (1 + *nexon + *nseg));
> for (int exon = 0; exon < *nexon; ++exon) {
> lagrangian[Nstart + 1 + exon] += lambda[(samp * *nexon) + exon];
> }
> }
>
> for (int entry = 0; entry < *nvar; ++entry) { current_X[entry] =
> X[entry] ; }
> *current_obj_factor = obj_factor;
> for (int entry = 0; entry < *ncon; ++entry) { current_lambda[entry] =
> lambda[entry] ; }
> for (int entry = 0; entry < *hess_entries; ++entry) {
> current_lagrangian[entry] = lagrangian[entry] ; }
> eval_counts[6] += 1;
>
> return true;
>
> }
>
>
> bool eval_ll_py(numeric::array const &X_py, numeric::array const &eval_py)
>
> {
>
> double *X = new double [*nvar];
> double eval;
>
> for (int entry = 0; entry < *nvar; ++entry) { X[entry] =
> double(X_py[entry]) ; }
>
> eval_ll(X, eval);
>
> eval_py[0] = eval;
>
> return true;
>
> }
>
>
> bool eval_cons_py(numeric::array const &X_py, numeric::array const &cons_py)
>
> {
>
> double *X = new double [*nvar];
> double *cons = new double [*ncon];
>
> for (int entry = 0; entry < *nvar; ++entry) { X[entry] =
> double(X_py[entry]) ; }
>
> eval_cons(X, cons);
>
> for (int entry = 0; entry < *ncon; ++entry) { cons_py[entry] =
> cons[entry] ; }
>
> return true;
>
> }
>
>
> bool eval_ll_grad_py(numeric::array const &X_py, numeric::array const
> &grad_py)
>
> {
>
> double *X = new double [*nvar];
> double *grad = new double [*nvar];
>
> for (int entry = 0; entry < *nvar; ++entry) { X[entry] =
> double(X_py[entry]) ; }
>
> eval_ll_grad(X, grad);
>
> for (int entry = 0; entry < *nvar; ++entry) { grad_py[entry] =
> grad[entry] ; }
>
> return true;
>
> }
>
>
> bool eval_cons_jac_py(numeric::array const &X_py, numeric::array const
> &jac_py)
>
> {
>
> double *X = new double [*nvar];
> double *jac = new double [*jac_entries];
>
> for (int entry = 0; entry < *nvar; ++entry) { X[entry] =
> double(X_py[entry]) ; }
>
> eval_cons_jac(X, jac);
>
> for (int entry = 0; entry < *jac_entries; ++entry) { jac_py[entry] =
> jac[entry] ; }
>
> return true;
>
> }
>
>
> bool eval_ll_hess_py(numeric::array const &X_py, numeric::array const
> &hess_py)
>
> {
>
> double *X = new double [*nvar];
> double *hess = new double [*hess_entries];
>
> for (int entry = 0; entry < *nvar; ++entry) { X[entry] =
> double(X_py[entry]) ; }
>
> eval_ll_hess(X, hess);
>
> for (int entry = 0; entry < *hess_entries; ++entry) { hess_py[entry] =
> hess[entry] ; }
>
> return true;
>
> }
>
>
> bool eval_cons_hess_py(numeric::array const &X_py, numeric::array const
> &hess_py)
>
> {
>
> double *X = new double [*nvar];
> double *hess = new double [*ncon];
>
> for (int entry = 0; entry < *nvar; ++entry) { X[entry] =
> double(X_py[entry]) ; }
>
> eval_cons_hess(X, hess);
>
> for (int entry = 0; entry < *ncon; ++entry) { hess_py[entry] =
> hess[entry] ; }
>
> return true;
>
> }
>
>
> bool eval_lagrangian_py(numeric::array const &X_py, numeric::array const
> &obj_factor_py, numeric::array const &lambda_py, numeric::array const
> &lagrangian_py)
>
> {
>
> double *X = new double [*nvar];
> double obj_factor;
> double *lambda = new double [*ncon];
> double *lagrangian = new double [*hess_entries];
>
> for (int entry = 0; entry < *nvar; ++entry) { X[entry] =
> double(X_py[entry]) ; }
>
> obj_factor = double(obj_factor_py[0]);
>
> for (int entry = 0; entry < *ncon; ++entry) { lambda[entry] =
> double(lambda_py[entry]) ; }
>
> eval_lagrangian(X, obj_factor, lambda, lagrangian);
>
> for (int entry = 0; entry < *hess_entries; ++entry) {
> lagrangian_py[entry] = lagrangian[entry] ; }
>
> return true;
>
> }
>
> /* Variable access */
>
> template <class T> bool update_var(std::string var_name, T var_value)
>
> {
>
> if (var_name == "gene_count") { gene_count = var_value ; }
> if (var_name == "nstate") { *nstate = var_value ; }
> if (var_name == "nsamp") { *nsamp = var_value ; }
> if (var_name == "nexon") { *nexon = var_value ; }
> if (var_name == "nseg") { *nseg = var_value ; }
> if (var_name == "nwin") { *nwin = var_value ; }
> if (var_name == "nvar") { *nvar = var_value ; }
> if (var_name == "ncon") { *ncon = var_value ; }
> if (var_name == "nsingle") { *nsingle = var_value ; }
> if (var_name == "hsingle") { *hsingle = var_value ; }
> if (var_name == "jac_entries") { *jac_entries = var_value ; }
> if (var_name == "hess_entries") { *hess_entries = var_value ; }
> if (var_name == "current_ll") { *current_ll = var_value ; }
> if (var_name == "current_obj_factor") { *current_obj_factor = var_value ; }
>
> }
>
> template <class T> bool return_var(std::string var_name, T &var_value)
>
> {
>
> if (var_name == "gene_count") {var_value = gene_count ; }
> if (var_name == "nstate") { var_value = *nstate ; }
> if (var_name == "nsamp") { var_value = *nsamp ; }
> if (var_name == "nexon") { var_value = *nexon ; }
> if (var_name == "nseg") { var_value = *nseg ; }
> if (var_name == "nwin") { var_value = *nwin ; }
> if (var_name == "nvar") { var_value = *nvar ; }
> if (var_name == "ncon") { var_value = *ncon ; }
> if (var_name == "nsingle") { var_value = *nsingle ; }
> if (var_name == "hsingle") { var_value = *hsingle ; }
> if (var_name == "jac_entries") { var_value = *jac_entries ; }
> if (var_name == "hess_entries") { var_value = *hess_entries ; }
> if (var_name == "current_ll") { var_value = *current_ll ; }
> if (var_name == "current_obj_factor") { var_value = *current_obj_factor ; }
>
>
> }
>
> template <class T> bool update_vec_var(std::string var_name, int
> var_length, T var_value[])
>
> {
>
> if (var_name == "statecols") {for (int entry = 0; entry < var_length;
> ++entry) { statecols[entry] = var_value[entry] ; } }
> if (var_name == "sampcols") {for (int entry = 0; entry < var_length;
> ++entry) { sampcols[entry] = var_value[entry] ; } }
> if (var_name == "exonrows") {for (int entry = 0; entry < var_length;
> ++entry) { exonrows[entry] = var_value[entry] ; } }
> if (var_name == "segrows") {for (int entry = 0; entry < var_length;
> ++entry) { segrows[entry] = var_value[entry] ; } }
> if (var_name == "sampsperstate") {for (int entry = 0; entry <
> var_length; ++entry) { sampsperstate[entry] = var_value[entry] ; } }
> if (var_name == "segsperexon") {for (int entry = 0; entry < var_length;
> ++entry) { segsperexon[entry] = var_value[entry] ; } }
> if (var_name == "single_hess_rows") {for (int entry = 0; entry <
> var_length; ++entry) { single_hess_rows[entry] = var_value[entry] ; } }
> if (var_name == "single_hess_cols") {for (int entry = 0; entry <
> var_length; ++entry) { single_hess_cols[entry] = var_value[entry] ; } }
> if (var_name == "jac_rows") {for (int entry = 0; entry < var_length;
> ++entry) { jac_rows[entry] = var_value[entry] ; } }
> if (var_name == "jac_cols") {for (int entry = 0; entry < var_length;
> ++entry) { jac_cols[entry] = var_value[entry] ; } }
> if (var_name == "full_hess_rows") {for (int entry = 0; entry <
> var_length; ++entry) { full_hess_rows[entry] = var_value[entry] ; } }
> if (var_name == "full_hess_cols") {for (int entry = 0; entry <
> var_length; ++entry) { full_hess_cols[entry] = var_value[entry] ; } }
> if (var_name == "reads") {for (int entry = 0; entry < var_length;
> ++entry) { reads[entry] = var_value[entry] ; } }
> if (var_name == "readmaxes") {for (int entry = 0; entry < var_length;
> ++entry) { readmaxes[entry] = var_value[entry] ; } }
> if (var_name == "eval_counts") {for (int entry = 0; entry < var_length;
> ++entry) { eval_counts[entry] = var_value[entry] ; } }
> if (var_name == "start_X") {for (int entry = 0; entry < var_length;
> ++entry) { start_X[entry] = var_value[entry] ; } }
> if (var_name == "x_L") {for (int entry = 0; entry < var_length; ++entry)
> { x_L[entry] = var_value[entry] ; } }
> if (var_name == "x_U") {for (int entry = 0; entry < var_length; ++entry)
> { x_U[entry] = var_value[entry] ; } }
> if (var_name == "g_L") {for (int entry = 0; entry < var_length; ++entry)
> { g_L[entry] = var_value[entry] ; } }
> if (var_name == "g_U") {for (int entry = 0; entry < var_length; ++entry)
> { g_U[entry] = var_value[entry] ; } }
> if (var_name == "current_X") {for (int entry = 0; entry < var_length;
> ++entry) { current_X[entry] = var_value[entry] ; } }
> if (var_name == "current_cons") {for (int entry = 0; entry < var_length;
> ++entry) { current_cons[entry] = var_value[entry] ; } }
> if (var_name == "current_ll_grad") {for (int entry = 0; entry <
> var_length; ++entry) { current_ll_grad[entry] = var_value[entry] ; } }
> if (var_name == "current_cons_jac") {for (int entry = 0; entry <
> var_length; ++entry) { current_cons_jac[entry] = var_value[entry] ; } }
> if (var_name == "current_ll_hess") {for (int entry = 0; entry <
> var_length; ++entry) { current_ll_hess[entry] = var_value[entry] ; } }
> if (var_name == "current_cons_hess") {for (int entry = 0; entry <
> var_length; ++entry) { current_cons_hess[entry] = var_value[entry] ; } }
> if (var_name == "current_lambda") {for (int entry = 0; entry <
> var_length; ++entry) { current_lambda[entry] = var_value[entry] ; } }
> if (var_name == "current_lagrangian") {for (int entry = 0; entry <
> var_length; ++entry) { current_lagrangian[entry] = var_value[entry] ; } }
>
> }
>
> template <class T> bool return_vec_var(std::string var_name, int
> var_length, T var_value[])
>
> {
>
> if (var_name == "statecols") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = statecols[entry] ; } }
> if (var_name == "sampcols") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = sampcols[entry] ; } }
> if (var_name == "exonrows") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = exonrows[entry] ; } }
> if (var_name == "segrows") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = segrows[entry] ; } }
> if (var_name == "sampsperstate") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = sampsperstate[entry] ; } }
> if (var_name == "segsperexon") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = segsperexon[entry] ; } }
> if (var_name == "single_hess_rows") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = single_hess_rows[entry] ; } }
> if (var_name == "single_hess_cols") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = single_hess_cols[entry] ; } }
> if (var_name == "jac_rows") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = jac_rows[entry] ; } }
> if (var_name == "jac_cols") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = jac_cols[entry] ; } }
> if (var_name == "full_hess_rows") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = full_hess_rows[entry] ; } }
> if (var_name == "full_hess_cols") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = full_hess_cols[entry] ; } }
> if (var_name == "reads") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = reads[entry] ; } }
> if (var_name == "readmaxes") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = readmaxes[entry] ; } }
> if (var_name == "eval_counts") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = eval_counts[entry] ; } }
> if (var_name == "start_X") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = start_X[entry] ; } }
> if (var_name == "x_L") {for (int entry = 0; entry < var_length; ++entry)
> { var_value[entry] = x_L[entry] ; } }
> if (var_name == "x_U") {for (int entry = 0; entry < var_length; ++entry)
> { var_value[entry] = x_U[entry] ; } }
> if (var_name == "g_L") {for (int entry = 0; entry < var_length; ++entry)
> { var_value[entry] = g_L[entry] ; } }
> if (var_name == "g_U") {for (int entry = 0; entry < var_length; ++entry)
> { var_value[entry] = g_U[entry] ; } }
> if (var_name == "current_X") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = current_X[entry] ; } }
> if (var_name == "current_cons") {for (int entry = 0; entry < var_length;
> ++entry) { var_value[entry] = current_cons[entry] ; } }
> if (var_name == "current_ll_grad") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = current_ll_grad[entry] ; } }
> if (var_name == "current_cons_jac") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = current_cons_jac[entry] ; } }
> if (var_name == "current_ll_hess") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = current_ll_hess[entry] ; } }
> if (var_name == "current_cons_hess") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = current_cons_hess[entry] ; } }
> if (var_name == "current_lambda") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = current_lambda[entry] ; } }
> if (var_name == "current_lagrangian") {for (int entry = 0; entry <
> var_length; ++entry) { var_value[entry] = current_lagrangian[entry] ; } }
>
> }
>
> bool update_var_py(std::string var_name_py, numeric::array const &const
> &var_value_py, int var_length_py, std::string var_type_py)
>
> {
>
> if (var_type_py == "int") {
> if (var_length_py == 1) {
> int var_value = int(var_value_py[0]);
> update_var(var_name_py, var_value);
> } else {
> int *var_value = new int [var_length_py];
> for (int entry = 0; entry < var_length_py; ++entry) {
> var_value[entry] = int(var_value_py[entry]);
> }
> update_vec_var(var_name_py, var_length_py, var_value);
> }
> } else {
> if (var_length_py == 1) {
> double var_value = double(var_value_py[0]);
> update_var(var_name_py, var_value);
> } else {
> double *var_value = new double [var_length_py];
> for (int entry = 0; entry < var_length_py; ++entry) {
> var_value[entry] = double(var_value_py[entry]);
> }
> update_vec_var(var_name_py, var_length_py, var_value);
> }
> }
>
> return true;
>
> }
>
>
> numeric::array const &return_var_py(std::string var_name_py, int
> var_length_py, std::string var_type_py)
>
> {
> list var_value_list_py = list();
>
> if (var_type_py == "int") {
> if (var_length_py == 1) {
> int var_value;
> return_var(var_name_py, var_value);
> var_value_py.append(int(var_value));
> } else {
> int *var_value = new int [var_length_py];
> return_vec_var(var_name_py, var_length_py, var_value);
> for (int entry = 0; entry < var_length_py; ++entry) {
> var_value_py.append(int(var_value[entry]));
> }
> }
> } else {
> if (var_length_py == 1) {
> double var_value;
> return_var(var_name_py, var_value);
> var_value_py.append(double(var_value));
> } else {
> double *var_value = new double [var_length_py];
> return_vec_var(var_name_py, var_length_py, var_value);
> for (int entry = 0; entry < var_length_py; ++entry) {
> var_value_py.append(double(var_value[entry]));
> }
> }
> }
>
> numeric::array const &var_value_array_py =
> numeric::array(var_value_list_py);
>
> return var_value_array_py;
>
> }
>
>
> private:
>
> int gene_count;
>
> boost::shared_ptr<int> nstate;
> boost::shared_ptr<int> nsamp;
> boost::shared_ptr<int> nexon;
> boost::shared_ptr<int> nseg;
> boost::shared_ptr<int> nwin;
> boost::shared_ptr<int> nvar;
> boost::shared_ptr<int> ncon;
> boost::shared_ptr<int> nsingle;
> boost::shared_ptr<int> hsingle;
> boost::shared_ptr<int> jac_entries;
> boost::shared_ptr<int> hess_entries;
>
> boost::shared_ptr<double> current_ll;
> boost::shared_ptr<double> current_obj_factor;
>
> boost::shared_array<int> statecols;
> boost::shared_array<int> sampcols;
> boost::shared_array<int> exonrows;
> boost::shared_array<int> segrows;
> boost::shared_array<int> sampsperstate;
> boost::shared_array<int> segsperexon;
> boost::shared_array<int> single_hess_rows;
> boost::shared_array<int> single_hess_cols;
> boost::shared_array<int> jac_rows;
> boost::shared_array<int> jac_cols;
> boost::shared_array<int> full_hess_rows;
> boost::shared_array<int> full_hess_cols;
> boost::shared_array<int> eval_counts;
>
> boost::shared_array<double> reads;
> boost::shared_array<double> readmaxes;
> boost::shared_array<double> start_X;
> boost::shared_array<double> x_L;
> boost::shared_array<double> x_U;
> boost::shared_array<double> g_L;
> boost::shared_array<double> g_U;
> boost::shared_array<double> current_X;
> boost::shared_array<double> current_cons;
> boost::shared_array<double> current_ll_grad;
> boost::shared_array<double> current_cons_jac;
> boost::shared_array<double> current_ll_hess;
> boost::shared_array<double> current_cons_hess;
> boost::shared_array<double> current_lambda;
> boost::shared_array<double> current_lagrangian;
>
>
> };
>
>
> }
>
>
> }
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig


From talljimbo at gmail.com  Sun May 13 00:48:09 2012
From: talljimbo at gmail.com (Jim Bosch)
Date: Sat, 12 May 2012 18:48:09 -0400
Subject: [C++-sig] How can I make myself usefull?
In-Reply-To: <4fa8b9df.e539320a.654d.42c9@mx.google.com>
References: <4fa8b9df.e539320a.654d.42c9@mx.google.com>
Message-ID: <4FAEE8A9.50301@gmail.com>

On 05/08/2012 02:14 AM, Matthew Scouten wrote:
> Hello,
> 	I have recently been laid off from a job where a made a lot of good
> use of Boost.Python. I would like to give something back (while incidentally
> keeping my skills from getting rusty). I have already been following the
> boost-python tag on stackoverflow and answering a lot of question there
> (http://stackoverflow.com/users/8508/matthew-scouten). I haven't hacked on
> the internals of BP much, but now that I have time.
>
> While I was there, I had some code (now lost to me) that made BP easier to
> use.
> Here is some of that I had:
>
> * A deepcopyable suite, so that any c++ class with an appropriate copy ctor
> could be quickly given a __deepcopy__, a __copy__ and a copying __init__,
> with a single line.
>
> * A similar compare suite, so that classes with == and<  could be given a
> full set of comparison operators, with a single line
>
> * 2 function templates, SafePointer2Object and SafeObject2Pointer which
> dealt with conversions between bp::objects that might be None and  pointers
> to c++ classes that might be NULL
>
> * Simple RAII objects that dealt with acquiring and freeing the GIL around
> callbacks on different threads. A similar one for freeing the GIL around a
> code block. An idea that maybe this could be a call policy.
>
> * No_compare_indexing_suite is vector_indexing_suite for classes without ==
>
> I would like to recreate some of these up for inclusion in BP, if you are
> interested. Or if there is other work that needs doing....

All of these sound very useful, and I'd love to see any of them in 
Boost.Python (well, to be honest, I don't use threading much, so I don't 
have much need for the GIL tools, but I know others on this list seem to 
be GIL-wrangling all the time).

Another thing to consider is C++11 support, and particularly support for 
the C++11 standard library.  I don't know think any of the original 
library builders or maintainers has any plans to devote any of their own 
time to add support for e.g. std::tuple or even std::shared_ptr and 
unique_ptr, but that's going to be increasingly important as C++11 
compiler support gets better.  Being able to wrap C++ functions that 
take rvalue reference arguments would be another important C++11 feature 
to support, but possibly a lot harder.

Thanks!

Jim

From micdestefano at gmail.com  Sun May 13 11:18:40 2012
From: micdestefano at gmail.com (Michele De Stefano)
Date: Sun, 13 May 2012 11:18:40 +0200
Subject: [C++-sig] Question on the development of Boost Python
Message-ID: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>

I've read a recent post by Jim Bosch saying that the original developers of
Boost Python are not going to upgrade it for supporting C++11 standard
libraries.

So I simply would like to understand if this great library is going to die
or if it is kept up to date by someone else.

Who are the current maintainers/developers?

Jim, are you one of the official developers (I ask because I've not found
your name into the Boost Python documentation)?

Thank you in advance.
Best regards.

-- 
Michele De Stefano
Web Site <http://www.micheledestefano.joomlafree.it>
Linked In <http://it.linkedin.com/in/micheledestefano>
mds-utils: a general purpose Open Source
library<http://code.google.com/p/mds-utils>
 <http://it.linkedin.com/in/micheledestefano>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120513/e6f363a4/attachment.html>

From talljimbo at gmail.com  Sun May 13 15:51:19 2012
From: talljimbo at gmail.com (Jim Bosch)
Date: Sun, 13 May 2012 09:51:19 -0400
Subject: [C++-sig] Question on the development of Boost Python
In-Reply-To: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>
References: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>
Message-ID: <4FAFBC57.3070003@gmail.com>

On 05/13/2012 05:18 AM, Michele De Stefano wrote:
> I've read a recent post by Jim Bosch saying that the original developers
> of Boost Python are not going to upgrade it for supporting C++11
> standard libraries.
>
> So I simply would like to understand if this great library is going to
> die or if it is kept up to date by someone else.
>
> Who are the current maintainers/developers?
>
> Jim, are you one of the official developers (I ask because I've not
> found your name into the Boost Python documentation)?
>

I'm not, and my assertion about their plans should be read as just an 
observation by someone who has been paying attention to the mailing list 
and codebase.

My sense is that critical errors and bugs are still being fixed, but 
there aren't many...and while patches are still being accepted when 
contributed, no one is really orchestrating development of new features. 
  But I'm hoping someone more authoritative will chime in on that subject.

A while ago I started some conversations about working on a major 
overhaul (i.e. a Boost.Python v3).  I got a lot of useful ideas, and I'm 
been working on it since - but that's been proceeding extremely slowly 
due to other commitments.  And it's turning into enough of a rewrite 
(I'm leaning towards making it a C++11 only library) that it's probably 
best to decouple those plans from improvements to the Boost.Python we 
have now.

Jim

From s_sourceforge at nedprod.com  Sun May 13 19:53:26 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Sun, 13 May 2012 18:53:26 +0100
Subject: [C++-sig] How can I make myself usefull?
Message-ID: <4FAFF516.25607.60BAE7B6@s_sourceforge.nedprod.com>

On 12 May 2012 at 18:48, Jim Bosch wrote:

> Another thing to consider is C++11 support, and particularly support for 
> the C++11 standard library.  I don't know think any of the original 
> library builders or maintainers has any plans to devote any of their own 
> time to add support for e.g. std::tuple or even std::shared_ptr and 
> unique_ptr, but that's going to be increasingly important as C++11 
> compiler support gets better.  Being able to wrap C++ functions that 
> take rvalue reference arguments would be another important C++11 feature 
> to support, but possibly a lot harder.

Very good point Jim. I must get round to reading the C++11 spec from 
cover to cover, it's not an exciting task but something which every 
C++ programmer ought to do. In particular I know I'm not familiar 
enough with the new library additions.  

Does anyone know of a good book summarising the new bits in C++11, or 
of such a book to be released in the next six months? A lot of C++ 
books aren't great in my experience, either badly written or too hand 
holding. Something to the point like "Modern C++ Design" would be 
great, but I can't see Andrei targeting C++11 now he has D to think 
about. I would be happy to learn otherwise though.  

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/



-------------- next part --------------
A non-text attachment was scrubbed...
Name: WPM$SM1H.PM$
Type: application/octet-stream
Size: 1273 bytes
Desc: Mail message body
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120513/aa62f74d/attachment.obj>

From stefan at seefeld.name  Sun May 13 20:31:55 2012
From: stefan at seefeld.name (Stefan Seefeld)
Date: Sun, 13 May 2012 14:31:55 -0400
Subject: [C++-sig] Question on the development of Boost Python
In-Reply-To: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>
References: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>
Message-ID: <4FAFFE1B.1040500@seefeld.name>

On 05/13/2012 05:18 AM, Michele De Stefano wrote:
> I've read a recent post by Jim Bosch saying that the original
> developers of Boost Python are not going to upgrade it for supporting
> C++11 standard libraries.

Can you elaborate a little on what you mean by "supporting C++ 11" ?
Specifically, in what way is it not supported right now ?

>
> So I simply would like to understand if this great library is going to
> die or if it is kept up to date by someone else.

Implying that it is "going to die" simply because it isn't changed to
support C++ 11 features is quite a stretch, don't you think ?

How many compilers exist right now that fully support C++ 11 ? And how
many users exist who are willing to switch at this point in time ?

    Stefan


-- 

      ...ich hab' noch einen Koffer in Berlin...


From micdestefano at gmail.com  Sun May 13 21:28:02 2012
From: micdestefano at gmail.com (Michele De Stefano)
Date: Sun, 13 May 2012 21:28:02 +0200
Subject: [C++-sig] Question on the development of Boost Python
In-Reply-To: <4FAFFE1B.1040500@seefeld.name>
References: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>
	<4FAFFE1B.1040500@seefeld.name>
Message-ID: <CAJ7vBf2dVnpMUms3u7CCpr=tPGuXJdf2_NsPAOPjT+Jb9qx_cg@mail.gmail.com>

I'm sorry,

I understand that with my argumentation I've given the wrong idea of what I
meant.

I read that post from Jim Bosch and I thought he knew something more about
Boost Python development.

Then I completely agree that a library is absolutely not dead also if it
does not support C++ 11. Personally, I won't be able to use C++ 11 for
years starting from now.

I would simply like to be re-ensured that Boost Python development is going
on, because I like this library a lot.

Best regards to all.
Michele

2012/5/13 Stefan Seefeld <stefan at seefeld.name>

> On 05/13/2012 05:18 AM, Michele De Stefano wrote:
> > I've read a recent post by Jim Bosch saying that the original
> > developers of Boost Python are not going to upgrade it for supporting
> > C++11 standard libraries.
>
> Can you elaborate a little on what you mean by "supporting C++ 11" ?
> Specifically, in what way is it not supported right now ?
>
> >
> > So I simply would like to understand if this great library is going to
> > die or if it is kept up to date by someone else.
>
> Implying that it is "going to die" simply because it isn't changed to
> support C++ 11 features is quite a stretch, don't you think ?
>
> How many compilers exist right now that fully support C++ 11 ? And how
> many users exist who are willing to switch at this point in time ?
>
>    Stefan
>
>
> --
>
>      ...ich hab' noch einen Koffer in Berlin...
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>



-- 
Michele De Stefano
Web Site <http://www.micheledestefano.joomlafree.it>
Linked In <http://it.linkedin.com/in/micheledestefano>
mds-utils: a general purpose Open Source
library<http://code.google.com/p/mds-utils>
<http://it.linkedin.com/in/micheledestefano>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120513/4686e5a8/attachment.html>

From stefan at seefeld.name  Sun May 13 21:45:52 2012
From: stefan at seefeld.name (Stefan Seefeld)
Date: Sun, 13 May 2012 15:45:52 -0400
Subject: [C++-sig] Question on the development of Boost Python
In-Reply-To: <CAJ7vBf2dVnpMUms3u7CCpr=tPGuXJdf2_NsPAOPjT+Jb9qx_cg@mail.gmail.com>
References: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>
	<4FAFFE1B.1040500@seefeld.name>
	<CAJ7vBf2dVnpMUms3u7CCpr=tPGuXJdf2_NsPAOPjT+Jb9qx_cg@mail.gmail.com>
Message-ID: <4FB00F70.1040902@seefeld.name>

On 05/13/2012 03:28 PM, Michele De Stefano wrote:
> I would simply like to be re-ensured that Boost Python development is
> going on, because I like this library a lot.

You are not alone ! :-)

    Stefan

-- 

      ...ich hab' noch einen Koffer in Berlin...


From j.wielicki at sotecware.net  Mon May 14 09:51:01 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Mon, 14 May 2012 09:51:01 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++ c++0x/11
 (testcase attached)
Message-ID: <4FB0B965.1050405@sotecware.net>

Moin folks,

I recently upgraded to boost::python 1.48 and g++ 4.7 (fedora 17). This
broke a project of mine as it crashes when passing an object with
virtual inheritance from python back to C++ (e.g. during a method call).
A minimal test case is attached.

After nearly an hour of debugging in a single gdb session, I found that
the value in the PyObject* itself does not change.
When the value was first wrapped into a PyObject*, I noted down the
address of that and did trial&error dereferencing until I was able to
get the actual wrapped object:

(gdb) disp (((PyUni::X11Window**)((long*)raw_result)[5])[2])

in the context of
/usr/include/boost/python/object/make_instance.hpp:49
(PyUni::X11Window* being the object to be wrapped)

The offset 5 was found by inspecting python's object.h, the 2 by
guessing (at [5] we find an auto_ptr instance).

I added a watchpoint to gdb on that location which did not trigger.
Also, after the segfault, I inspected the same adress by replacing
raw_result with the appropriate pointer (I noted down the address
raw_result was pointing to before continuing from make_instance.hpp:49).


Then I was able to reproduce the problem and pin it down to virtual
inheritance and g++ -std=c++0x (c++11 does the same), so I built the
attached test case. The archive contains (no tarbomb) a single cpp file
and two methods to build it. For those for whom it works, there is the
simple.Makefile, which relies on hardcoded paths. A CMakeLists.txt is
provided for those with cmake and for whom the simple Makefile does not
work.

Another quick test with boost 1.49 did not fix the issue.

ps: I hope this is not duped now, I accidentially used the wrong sender
address first.

-- 
Jonas

-------------- next part --------------
A non-text attachment was scrubbed...
Name: bpcrash.tar.gz
Type: application/x-gzip
Size: 1247 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120514/05797f9a/attachment.bin>

From j.wielicki at sotecware.net  Mon May 14 09:48:28 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Mon, 14 May 2012 09:48:28 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++ c++0x/11
 (testcase attached)
Message-ID: <4FB0B8CC.3050001@sotecware.net>

Moin folks,

I recently upgraded to boost::python 1.48 and g++ 4.7 (fedora 17). This
broke a project of mine as it crashes when passing an object with
virtual inheritance from python back to C++ (e.g. during a method call).
A minimal test case is attached.

After nearly an hour of debugging in a single gdb session, I found that
the value in the PyObject* itself does not change.
When the value was first wrapped into a PyObject*, I noted down the
address of that and did trial&error dereferencing until I was able to
get the actual wrapped object:

(gdb) disp (((PyUni::X11Window**)((long*)raw_result)[5])[2])

in the context of
/usr/include/boost/python/object/make_instance.hpp:49
(PyUni::X11Window* being the object to be wrapped)

The offset 5 was found by inspecting python's object.h, the 2 by
guessing (at [5] we find an auto_ptr instance).

I added a watchpoint to gdb on that location which did not trigger.
Also, after the segfault, I inspected the same adress by replacing
raw_result with the appropriate pointer (I noted down the address
raw_result was pointing to before continuing from make_instance.hpp:49).


Then I was able to reproduce the problem and pin it down to virtual
inheritance and g++ -std=c++0x (c++11 does the same), so I built the
attached test case. The archive contains (no tarbomb) a single cpp file
and two methods to build it. For those for whom it works, there is the
simple.Makefile, which relies on hardcoded paths. A CMakeLists.txt is
provided for those with cmake and for whom the simple Makefile does not
work.

Another quick test with boost 1.49 did not fix the issue.

-- 
Jonas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bpcrash.tar.gz
Type: application/x-gzip
Size: 1247 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120514/ed8f8286/attachment.bin>

From s_sourceforge at nedprod.com  Mon May 14 12:15:26 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 14 May 2012 11:15:26 +0100
Subject: [C++-sig] Question on the development of Boost Python
In-Reply-To: <CAJ7vBf2dVnpMUms3u7CCpr=tPGuXJdf2_NsPAOPjT+Jb9qx_cg@mail.gmail.com>
References: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>,
	<4FAFFE1B.1040500@seefeld.name>,
	<CAJ7vBf2dVnpMUms3u7CCpr=tPGuXJdf2_NsPAOPjT+Jb9qx_cg@mail.gmail.com>
Message-ID: <4FB0DB3E.32559.643EFC2E@s_sourceforge.nedprod.com>

On 13 May 2012 at 21:28, Michele De Stefano wrote:

> I read that post from Jim Bosch and I thought he knew something more about
> Boost Python development.

What Jim didn't mention is how BPL v2 (the current version) was first 
written. Basically, Berkeley paid Dave Abrahams a chunk of money to 
(re)write it and others have since bolted on extras, fixed bugs etc. 
But Dave designed and wrote the core.  

Dave, needing to put bread on the table, has to choose to only code 
where he is paid for it. If someone stumps up the cash to 
substantially improve BPL, I don't doubt Dave will be only too happy 
to fix all sorts of things in BPL. There is a long list of known 
issues in BPL. Unfortunately, the number of complainants (and I 
include myself here) on this mailing list has never matched the 
willingness to front cash or personal time to fix them. That's not a 
criticism, it's just how open source works.

> I would simply like to be re-ensured that Boost Python development is going
> on, because I like this library a lot.

The best way to ensure that Boost Python development goes on is to 
send a few thousand dollars per year to Dave and/or Boost Consulting 
for maintenance. Unfortunately, as far as I am aware, no one has done 
so for Boost Python in some years now which is a great shame as 
bitrot has definitely set into BPL. In the end though, if it's really 
important enough to people, someone will eventually either stump up 
cash or do the rennovations themselves.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From s_sourceforge at nedprod.com  Mon May 14 12:15:27 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 14 May 2012 11:15:27 +0100
Subject: [C++-sig] Question on the development of Boost Python
In-Reply-To: <4FAFFE1B.1040500@seefeld.name>
References: <CAJ7vBf1VAMCSQS7tkDjnb+RHimnwf-TaumYKm0kzYfJYLwc-jw@mail.gmail.com>,
	<4FAFFE1B.1040500@seefeld.name>
Message-ID: <4FB0DB3F.4046.643EFCCA@s_sourceforge.nedprod.com>

On 13 May 2012 at 14:31, Stefan Seefeld wrote:

> On 05/13/2012 05:18 AM, Michele De Stefano wrote:
> > I've read a recent post by Jim Bosch saying that the original
> > developers of Boost Python are not going to upgrade it for supporting
> > C++11 standard libraries.
> 
> Can you elaborate a little on what you mean by "supporting C++ 11" ?
> Specifically, in what way is it not supported right now ?

Feeding some forms of code using C++11 into BPL would cause it to 
choke.

> > So I simply would like to understand if this great library is going to
> > die or if it is kept up to date by someone else.
> 
> Implying that it is "going to die" simply because it isn't changed to
> support C++ 11 features is quite a stretch, don't you think ?

"Bitrot" is a more appropriate term.

> How many compilers exist right now that fully support C++ 11 ? And how
> many users exist who are willing to switch at this point in time ?

I've been using bits of C++11 in my own code since 2004 :) yes, they 
were all #ifdef'd out according to a compiler environment test, but I 
wrote in support for rvalue refs from the very beginning. It was 
extremely pleasing when the very first GCC to support rvalues became 
available and I could finally compile that code (needless to say, 
none of it worked as the spec had changed).

A lot of users are using small bits of C++11 in their C++. Variadic 
templates alone just make things so very much faster, and static 
assertions are a boon as is noexcept instead of throw(). I personally 
don't use the new container iteration for loop construct yet, but I 
am liking the new use of the auto keyword in the old form contaner 
iteration loop.

My point is that no one ever switches. Rather, conveniences creep in, 
and less people raise bugs when an older compiler no longer compiles. 
One day everyone is no longer targeting the old standard. It doesn't 
mean, of course, that they're targeting the new standard either :)

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From s_sourceforge at nedprod.com  Mon May 14 12:37:58 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 14 May 2012 11:37:58 +0100
Subject: [C++-sig] boost::python with virtual inheritance and g++
	c++0x/11 (testcase attached)
In-Reply-To: <4FB0B8CC.3050001@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>
Message-ID: <4FB0E086.24110.64539BB9@s_sourceforge.nedprod.com>

On 14 May 2012 at 9:48, Jonas Wielicki wrote:

> I recently upgraded to boost::python 1.48 and g++ 4.7 (fedora 17). This
> broke a project of mine as it crashes when passing an object with
> virtual inheritance from python back to C++ (e.g. during a method call).
> A minimal test case is attached.

I would be almost certain that your bug stems from changes to GCC, 
not changes to BPL. Generally speaking GCC bugs fix themselves over 
time, so you can revert temporarily to an older GCC. Bugs in GCC are 
common enough that if my code fails, I first think it's GCC's fault 
:)

Try an older GCC. Also, try Clang. Clang seems to generate better 
debug info for templates anyway.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From j.wielicki at sotecware.net  Mon May 14 12:53:32 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Mon, 14 May 2012 12:53:32 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FB0E086.24110.64539BB9@s_sourceforge.nedprod.com>
References: <4FB0B8CC.3050001@sotecware.net>
	<4FB0E086.24110.64539BB9@s_sourceforge.nedprod.com>
Message-ID: <4FB0E42C.7050308@sotecware.net>

On 14/05/12 12:37, Niall Douglas wrote:
> I would be almost certain that your bug stems from changes to GCC, 
> not changes to BPL. 
By now, this is also my suspect. Mainly, because it should be just one
function call between the value being correct and being wrong, while it
is only passed as a by-value argument.

> Generally speaking GCC bugs fix themselves over 
> time, so you can revert temporarily to an older GCC. Bugs in GCC are 
> common enough that if my code fails, I first think it's GCC's fault 
> :)
Hm. GCC updates are rather rare though I think? Also its kinda
inconvienient to use a non-distribution gcc version. Do you recommend to
submit a bug report to the gcc folks on that one?

> Also, try Clang. Clang seems to generate better 
> debug info for templates anyway.
I'll have a look, thanks.

-- 
Jonas

From spartan811 at hotmail.com  Mon May 14 13:28:57 2012
From: spartan811 at hotmail.com (Yoann Chaumy)
Date: Mon, 14 May 2012 13:28:57 +0200
Subject: [C++-sig] how include library to Jamroot
In-Reply-To: <CAAAfPCDAiwJGOtYi6bKbg_jOJgZBuv8-Lz=jSZESM0pzJHF=9g@mail.gmail.com>
References: <CAAAfPCCVzw4YxN=QqnfyzJ3xuVGHVU6KOk6scQdtgRMaveJMOQ@mail.gmail.com>,
	<4F9AB87D.7070404@gmail.com>,
	<CAAAfPCDAiwJGOtYi6bKbg_jOJgZBuv8-Lz=jSZESM0pzJHF=9g@mail.gmail.com>
Message-ID: <BLU158-W208F96A9A1E91E3DDA9BE59C1A0@phx.gbl>


 Hello,  I using bjam to compile my C++ program with Boost.Python But I have a link error : error LNK2019: unresolved external symbol "__declspec(dllimport).... so I try to add my library in the Jamroot file :  project
  : requirements <library>/boost/python//boost_python 
    <library-path>D:/localExt_win32_cl10-0Exp/lib    <library-path>D:/local/bin    <include> ..... It that right ?  thanks 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120514/82ed2d45/attachment.html>

From s_sourceforge at nedprod.com  Mon May 14 13:33:27 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 14 May 2012 12:33:27 +0100
Subject: [C++-sig] boost::python with virtual inheritance and g++
	c++0x/11 (testcase attached)
In-Reply-To: <4FB0E42C.7050308@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FB0E086.24110.64539BB9@s_sourceforge.nedprod.com>,
	<4FB0E42C.7050308@sotecware.net>
Message-ID: <4FB0ED87.23804.64866704@s_sourceforge.nedprod.com>

On 14 May 2012 at 12:53, Jonas Wielicki wrote:
> > Generally speaking GCC bugs fix themselves over 
> > time, so you can revert temporarily to an older GCC. Bugs in GCC are 
> > common enough that if my code fails, I first think it's GCC's fault 
> > :)
> Hm. GCC updates are rather rare though I think?

Not at all - GCC has a very rapid release cycle compared to most 
other C++ compilers. Most compilers are only released every two or 
even four years.

> Also its kinda
> inconvienient to use a non-distribution gcc version.

Most distros bundle multiple versions of gcc due to this exact 
problem with frequent bugs. Debian, at least, usually has the 
previous version and for many years also had the latest from the 3.x 
and indeed 2.x series.

Simply have your build chain wire in a non-default compiler by 
excluding from a list of known broken gcc's. It's what my build 
system does.

> Do you recommend to
> submit a bug report to the gcc folks on that one?

I personally do. But getting a bug accepted by gcc isn't easy. You 
may need to be prepared for them to be a bit caustic to you.

A very small testcase is easiest. They don't like testcases bringing 
in external libraries such as boost one bit. So be prepared to be 
aggressive if needs be.

> > Also, try Clang. Clang seems to generate better 
> > debug info for templates anyway.
> I'll have a look, thanks.

Personally speaking now BSD has switched to Clang I think GCC is 
living on borrowed time. Clang can already build 90%+ of all Debian, 
and it won't be long before it's 100%. At that point there is no 
longer any good reason to continue using GCC except for the languages 
LLVM doesn't support.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From j.wielicki at sotecware.net  Mon May 14 15:11:36 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Mon, 14 May 2012 15:11:36 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FB0ED87.23804.64866704@s_sourceforge.nedprod.com>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FB0E086.24110.64539BB9@s_sourceforge.nedprod.com>,
	<4FB0E42C.7050308@sotecware.net>
	<4FB0ED87.23804.64866704@s_sourceforge.nedprod.com>
Message-ID: <4FB10488.3070802@sotecware.net>

On 05/14/2012 01:33 PM, Niall Douglas wrote:
> Most distros bundle multiple versions of gcc due to this exact 
> problem with frequent bugs. Debian, at least, usually has the 
> previous version and for many years also had the latest from the 3.x 
> and indeed 2.x series.
Well, Fedora doesn't. They only ship a 3.4 series compiler as an
alternative (maybe to build qt3 apps or something?). But actually I like
-std=C++0x, you know.

> Simply have your build chain wire in a non-default compiler by 
> excluding from a list of known broken gcc's. It's what my build 
> system does.
Hm. How did you implement that? I mean, does the build chain build its
own gcc if neccessary and install it to a well-defined path in the
user's home directory?

> On 14 May 2012 at 12:53, Jonas Wielicki wrote:
>> Do you recommend to
>> submit a bug report to the gcc folks on that one?
> 
> I personally do. But getting a bug accepted by gcc isn't easy. You 
> may need to be prepared for them to be a bit caustic to you.
I guessed so. Have to remind the discussion between linux kernel folks
and gcc people over whether its a bug in the gcc when a null pointer
derefenciation in the kernel opens a security hole ;).

> A very small testcase is easiest. They don't like testcases bringing 
> in external libraries such as boost one bit. So be prepared to be 
> aggressive if needs be.
Hm. I'm afraid that this bug is not easy to separate in a small
testcase, because if it was evident, the fedora folks would've (a) filed
a bug at the gcc or (b) put in a patch themselves but definetly (c) not
shipped that compiler.

>>> Also, try Clang. Clang seems to generate better 
>>> debug info for templates anyway.
>> I'll have a look, thanks.
Clang doesn't compile boost python 1.49 at all. Sadly.

I guess I'll have to temporarily build a 4.6 gcc for myself, until that
issue is fixed.

-- 
Jonas

From s_sourceforge at nedprod.com  Mon May 14 16:05:09 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 14 May 2012 15:05:09 +0100
Subject: [C++-sig] boost::python with virtual inheritance and g++
	c++0x/11 (testcase attached)
In-Reply-To: <4FB10488.3070802@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FB0ED87.23804.64866704@s_sourceforge.nedprod.com>,
	<4FB10488.3070802@sotecware.net>
Message-ID: <4FB11115.20377.651148E9@s_sourceforge.nedprod.com>

On 14 May 2012 at 15:11, Jonas Wielicki wrote:

> On 05/14/2012 01:33 PM, Niall Douglas wrote:
> > Most distros bundle multiple versions of gcc due to this exact 
> > problem with frequent bugs. Debian, at least, usually has the 
> > previous version and for many years also had the latest from the 3.x 
> > and indeed 2.x series.
> Well, Fedora doesn't. They only ship a 3.4 series compiler as an
> alternative (maybe to build qt3 apps or something?). But actually I like
> -std=C++0x, you know.

Haven't touched Fedora in a decade I'm afraid. I should clarify my 
earlier statement: Debian doesn't necessarily install by default 
multiple versions of compilers, rather they're in the repos available 
as package dependencies. Surely Fedora is the same?

> > Simply have your build chain wire in a non-default compiler by 
> > excluding from a list of known broken gcc's. It's what my build 
> > system does.
> Hm. How did you implement that? I mean, does the build chain build its
> own gcc if neccessary and install it to a well-defined path in the
> user's home directory?

Ah, my build system has been scons for years now. As that's really 
python, it's trivial to scan /usr/bin for anything which can compile 
and invoke each with --version or take a guess from their filename.

My build system simply warns with big letters if the only compiler 
available is known to be broken and suggests you should get a 
different one, but one can force proceed if you wish.

> > A very small testcase is easiest. They don't like testcases bringing 
> > in external libraries such as boost one bit. So be prepared to be 
> > aggressive if needs be.
> Hm. I'm afraid that this bug is not easy to separate in a small
> testcase, because if it was evident, the fedora folks would've (a) filed
> a bug at the gcc or (b) put in a patch themselves but definetly (c) not
> shipped that compiler.

Indeed. My experience has been trying when reporting complex 
metaprogramming bugs in GCC. They say things like "we regularly test 
against Boost and the compiler passes all its regression tests, 
therefore the bug can't be in our compiler". Understandable, but 
frustrating.

> >>> Also, try Clang. Clang seems to generate better 
> >>> debug info for templates anyway.
> >> I'll have a look, thanks.
> Clang doesn't compile boost python 1.49 at all. Sadly.

I am deeply surprised due to 
http://blog.llvm.org/2010/05/clang-builds-boost.html.

http://www.boost.org/development/tests/release/developer/python.html 
reports Clang to be no worse than MSVC, and much better than GCC 4.2.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From j.wielicki at sotecware.net  Mon May 14 16:22:54 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Mon, 14 May 2012 16:22:54 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FB11115.20377.651148E9@s_sourceforge.nedprod.com>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FB0ED87.23804.64866704@s_sourceforge.nedprod.com>,
	<4FB10488.3070802@sotecware.net>
	<4FB11115.20377.651148E9@s_sourceforge.nedprod.com>
Message-ID: <4FB1153E.5050406@sotecware.net>

On 05/14/2012 04:05 PM, Niall Douglas wrote:
> Debian doesn't necessarily install by default 
> multiple versions of compilers, rather they're in the repos available 
> as package dependencies. Surely Fedora is the same?
Nope, I was talking about the repositories.

> Ah, my build system has been scons for years now. As that's really 
> python, it's trivial to scan /usr/bin for anything which can compile 
> and invoke each with --version or take a guess from their filename.
Okay, we're on cmake here. I really should take a look at scons. I mean,
its python.

> Indeed. My experience has been trying when reporting complex 
> metaprogramming bugs in GCC. They say things like "we regularly test 
> against Boost and the compiler passes all its regression tests, 
> therefore the bug can't be in our compiler". Understandable, but 
> frustrating.
This sounds a bit like one should add that testcase to the regression
testsuite in boost to get a gcc bug fixed, eh? ;).

>> Clang doesn't compile boost python 1.49 at all. Sadly.
Sure, you're right. Sorry, it was my fault in reading clang's error
messages. It chockes on the c++ stl bits, not on boost, but they're
included from boost, so the first thing I read was ?boost?.
Well, I cannot tell if it would choke on boost too, but I guess your
testing setup is more sane than mine (at least it's not holding a spatula).

-- 
Jonas

From brandsmeier at gmx.de  Mon May 14 16:43:29 2012
From: brandsmeier at gmx.de (Holger Brandsmeier)
Date: Mon, 14 May 2012 16:43:29 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FB1153E.5050406@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>
	<4FB0ED87.23804.64866704@s_sourceforge.nedprod.com>
	<4FB10488.3070802@sotecware.net>
	<4FB11115.20377.651148E9@s_sourceforge.nedprod.com>
	<4FB1153E.5050406@sotecware.net>
Message-ID: <CAFP7zVtjabhE4tD8aL0B3585JyqJ94LtNuXww2soJLaK7_xKMg@mail.gmail.com>

>>> Clang doesn't compile boost python 1.49 at all. Sadly.
> Sure, you're right. Sorry, it was my fault in reading clang's error
> messages. It chockes on the c++ stl bits, not on boost, but they're
> included from boost, so the first thing I read was ?boost?.
> Well, I cannot tell if it would choke on boost too, but I guess your
> testing setup is more sane than mine (at least it's not holding a spatula).

That is a typical error with clang, which is typically solved by point 6 here:
  http://clang.llvm.org/get_started.html
To use that advice you have to build your own clang though (not
difficult at all). Clang by default uses gcc's system libraries to be
binary compatible to the rest of Linux. The difficult part is that
this has to be adjusted for every distribution and every gcc version.
I experienced problems with this for multiple distributions.

-Holger

From s_sourceforge at nedprod.com  Mon May 14 17:52:11 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 14 May 2012 16:52:11 +0100
Subject: [C++-sig] boost::python with virtual inheritance and g++
	c++0x/11 (testcase attached)
In-Reply-To: <4FB1153E.5050406@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FB11115.20377.651148E9@s_sourceforge.nedprod.com>,
	<4FB1153E.5050406@sotecware.net>
Message-ID: <4FB12A2B.15347.657347F3@s_sourceforge.nedprod.com>

On 14 May 2012 at 16:22, Jonas Wielicki wrote:

> > Debian doesn't necessarily install by default 
> > multiple versions of compilers, rather they're in the repos available 
> > as package dependencies. Surely Fedora is the same?
> Nope, I was talking about the repositories.

Well, well. Didn't think that likely.

> > Ah, my build system has been scons for years now. As that's really 
> > python, it's trivial to scan /usr/bin for anything which can compile 
> > and invoke each with --version or take a guess from their filename.
> Okay, we're on cmake here. I really should take a look at scons. I mean,
> its python.

scons suffers from an O(N^2) complexity with source file count. Other 
than that it's great ...

> >> Clang doesn't compile boost python 1.49 at all. Sadly.
> Sure, you're right. Sorry, it was my fault in reading clang's error
> messages. It chockes on the c++ stl bits, not on boost, but they're
> included from boost, so the first thing I read was "boost".
> Well, I cannot tell if it would choke on boost too, but I guess your
> testing setup is more sane than mine (at least it's not holding a spatula).

It chokes on the GCC STL sure. Not so much its own STL. I am out of 
date here though, it's been some years since I debugged C++ on POSIX 
as recently cygwin has spat out fully working binaries on Windows 
after being debugged in MSVC, so no need to go native POSIX recently.

I'm actually starting a new job with a well known multinational this 
autumn where I'll be developing using C++ on POSIX exclusively. That 
should bring me very much up to date. My first personal order of 
business, coincidentally, will be porting their build environment 
from GCC to Clang to gain some sanity for the metaprogramming that 
comes next.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From dominic.sacre at gmx.de  Wed May 16 02:54:21 2012
From: dominic.sacre at gmx.de (Dominic =?iso-8859-1?q?Sacr=E9?=)
Date: Wed, 16 May 2012 02:54:21 +0200
Subject: [C++-sig] To-/From-Python converters for shared_ptr types
Message-ID: <201205160254.21933.dominic.sacre@gmx.de>

Hi,

I have written custom converters from Python to C++ types and vice versa. 
Now I'd like to wrap C++ functions that expect/return shared pointers to 
those types.

I could of course write another set of converters for each type, but that 
would mean duplicating a lot of code for no good reason. Is there a simple 
way to generate converters for shared_ptr<T>, if converters for T already 
exist?

See below for a simple test case. From Python I can call test.foo(42) and 
get back the same value. Calling test.bar(42) fails, because there's no 
converter from int to shared_ptr<Test>. What's the easiest way to make 
bar() behave the same way as foo()?


Thanks in advance,

Dominic


--- test module ---

#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>

namespace bp = boost::python;

struct Test {
    int value;
};

struct test_from_python_converter {
    test_from_python_converter() {
        bp::converter::registry::push_back(
                &convertible, &construct, bp::type_id<Test>());
    }
    static void *convertible(PyObject *obj) {
        return PyInt_Check(obj) ? obj : 0;
    }
    static void construct(PyObject *obj,
                bp::converter::rvalue_from_python_stage1_data *data) {
        void *storage = (reinterpret_cast<bp::converter::
                rvalue_from_python_storage<Test>*>(data))->storage.bytes;
        new (storage) Test();
        ((Test *)storage)->value = bp::extract<int>(obj);
        data->convertible = storage;
    }
};

struct test_to_python_converter
  : bp::to_python_converter<Test, test_to_python_converter, true>
{
    static PyObject *convert(Test const & t) {
        return PyInt_FromLong(t.value);
    }
    static PyTypeObject const *get_pytype() {
        return &PyInt_Type;
    }
};

Test foo(Test t) {
    return t;
}

boost::shared_ptr<Test> bar(boost::shared_ptr<Test> pt) {
    return pt;
}

BOOST_PYTHON_MODULE(test) {
    test_from_python_converter();
    test_to_python_converter();

    bp::def("foo", &foo);
    bp::def("bar", &bar);
}

From gabe at gaberivescorbett.com  Thu May 17 23:42:52 2012
From: gabe at gaberivescorbett.com (Gabe Rives-Corbett)
Date: Thu, 17 May 2012 17:42:52 -0400
Subject: [C++-sig] Segmentation Fault with Boost.Python and Inheritance
Message-ID: <61DAADFBCA6D4C468EF8216B42EBAE48@gaberivescorbett.com>

Hello, 

I'm getting a seg fault when trying to call a virtual method on a base class from python.  Code is below:

#include <memory>

namespace boost { 
  template<class T> const T* get_pointer(const std::shared_ptr<T>& ptr) 
  {
    return ptr.get();
  }

  template<class T> T* get_pointer(std::shared_ptr<T>& ptr)
  {
    return ptr.get();
  }
}

#include <Python.h>
#include <boost/python.hpp>

namespace bp = boost::python;

class MyBase
{
public:
MyBase(){}
virtual ~MyBase(){}

virtual void baseTest() { printf("base test\n"); }
};

class MyDerived : public MyBase
{
public:
MyDerived() {}
virtual ~MyDerived(){}

void derivedTest() { printf("derived test\n"); }
};

BOOST_PYTHON_MODULE(PythonTest)
{
bp::class_<MyBase, std::shared_ptr<MyBase>>("MyBase")
.def("baseTest", &MyBase::baseTest);

bp::class_<MyDerived, bp::bases<MyBase>, std::shared_ptr<MyDerived>>("MyDerived")
.def("derivedTest", &MyDerived::derivedTest);

bp::implicitly_convertible<std::shared_ptr<MyDerived>, std::shared_ptr<MyBase>>();
}

Does it have to do with using std::shared_ptr for storage?  If so is there a way around this?  Any help is appreciated.   

-- 
Gabe Rives-Corbett

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120517/c42e81a8/attachment.html>

From j.wielicki at sotecware.net  Fri May 18 10:41:03 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Fri, 18 May 2012 10:41:03 +0200
Subject: [C++-sig] Segmentation Fault with Boost.Python and Inheritance
In-Reply-To: <61DAADFBCA6D4C468EF8216B42EBAE48@gaberivescorbett.com>
References: <61DAADFBCA6D4C468EF8216B42EBAE48@gaberivescorbett.com>
Message-ID: <4FB60B1F.2050708@sotecware.net>

On 17/05/12 23:42, Gabe Rives-Corbett wrote:
> I'm getting a seg fault when trying to call a virtual method on a base class from python.
I had a similar problem a few days ago. Which g++ version are you using?
I had problems with 4.7.0 (redhat) while 4.6.3 (redhat) was working fine.

-- 
Jonas

From s_sourceforge at nedprod.com  Fri May 18 19:47:46 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Fri, 18 May 2012 18:47:46 +0100
Subject: [C++-sig] Segmentation Fault with Boost.Python and Inheritance
In-Reply-To: <4FB60B1F.2050708@sotecware.net>
References: <61DAADFBCA6D4C468EF8216B42EBAE48@gaberivescorbett.com>,
	<4FB60B1F.2050708@sotecware.net>
Message-ID: <4FB68B42.31250.7A79E9DE@s_sourceforge.nedprod.com>

So it was a bug in GCC after all. Useful to know.

Niall


On 18 May 2012 at 10:41, Jonas Wielicki wrote:

> On 17/05/12 23:42, Gabe Rives-Corbett wrote:
> > I'm getting a seg fault when trying to call a virtual method on a base class from python.
> I had a similar problem a few days ago. Which g++ version are you using?
> I had problems with 4.7.0 (redhat) while 4.6.3 (redhat) was working fine.
> 
> -- 
> Jonas
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig


-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From j.wielicki at sotecware.net  Fri May 18 20:28:16 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Fri, 18 May 2012 20:28:16 +0200
Subject: [C++-sig] Segmentation Fault with Boost.Python and Inheritance
In-Reply-To: <4FB68B42.31250.7A79E9DE@s_sourceforge.nedprod.com>
References: <61DAADFBCA6D4C468EF8216B42EBAE48@gaberivescorbett.com>,
	<4FB60B1F.2050708@sotecware.net>
	<4FB68B42.31250.7A79E9DE@s_sourceforge.nedprod.com>
Message-ID: <4FB694C0.1050306@sotecware.net>

On 18.05.2012 19:47, Niall Douglas wrote:
> So it was a bug in GCC after all. Useful to know.
Sorry that I forgot to mention. I was pretty sure it was the GCC after I
tried with two different Boost versions and the very same code was
running fine on gcc 4.6.3 platforms. It is not 100% bulletproof, but I
guess its the GCCs fault.

-- 
Jonas

From s_sourceforge at nedprod.com  Fri May 18 20:37:08 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Fri, 18 May 2012 19:37:08 +0100
Subject: [C++-sig] Segmentation Fault with Boost.Python and Inheritance
In-Reply-To: <4FB694C0.1050306@sotecware.net>
References: <61DAADFBCA6D4C468EF8216B42EBAE48@gaberivescorbett.com>,
	<4FB68B42.31250.7A79E9DE@s_sourceforge.nedprod.com>,
	<4FB694C0.1050306@sotecware.net>
Message-ID: <4FB696D4.5445.7AA71B63@s_sourceforge.nedprod.com>

It's especially useful to know because this list can expect the same 
problem to be reported as a bug in BPL repeatedly from now on :)

Did you submit the bug to GCC's bugzilla, and if you did do you have 
a link?

If you didn't, submitting it and having it rejected is worth doing 
just so we can link to the bug from here.

Niall


On 18 May 2012 at 20:28, Jonas Wielicki wrote:

> On 18.05.2012 19:47, Niall Douglas wrote:
> > So it was a bug in GCC after all. Useful to know.
> Sorry that I forgot to mention. I was pretty sure it was the GCC after I
> tried with two different Boost versions and the very same code was
> running fine on gcc 4.6.3 platforms. It is not 100% bulletproof, but I
> guess its the GCCs fault.
> 
> -- 
> Jonas
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig


-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From gabe at gaberivescorbett.com  Sat May 19 00:43:39 2012
From: gabe at gaberivescorbett.com (Gabe Rives-Corbett)
Date: Fri, 18 May 2012 18:43:39 -0400
Subject: [C++-sig] Segmentation Fault with Boost.Python and Inheritance
In-Reply-To: <mailman.25.1337335202.6798.cplusplus-sig@python.org>
References: <mailman.25.1337335202.6798.cplusplus-sig@python.org>
Message-ID: <82586BA9580946A79BAE019F64BBC4EE@gaberivescorbett.com>

Yes I'm using GCC 4.7 on OS X.  Is there any possible workaround to use inheritance with GCC 4.7?  I'm currently using c++11 threads which aren't supported on GCC 4.6 on OS X.  

-- 
Gabe Rives-Corbett
Cell: (805) 570-8395


On Friday, May 18, 2012 at 6:00 AM, cplusplus-sig-request at python.org wrote:

> Send Cplusplus-sig mailing list submissions to
> cplusplus-sig at python.org (mailto:cplusplus-sig at python.org)
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mail.python.org/mailman/listinfo/cplusplus-sig
> or, via email, send a message with subject or body 'help' to
> cplusplus-sig-request at python.org (mailto:cplusplus-sig-request at python.org)
> 
> You can reach the person managing the list at
> cplusplus-sig-owner at python.org (mailto:cplusplus-sig-owner at python.org)
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Cplusplus-sig digest..."
> 
> 
> Today's Topics:
> 
> 1. Segmentation Fault with Boost.Python and Inheritance
> (Gabe Rives-Corbett)
> 2. Re: Segmentation Fault with Boost.Python and Inheritance
> (Jonas Wielicki)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Thu, 17 May 2012 17:42:52 -0400
> From: Gabe Rives-Corbett <gabe at gaberivescorbett.com (mailto:gabe at gaberivescorbett.com)>
> To: cplusplus-sig at python.org (mailto:cplusplus-sig at python.org)
> Subject: [C++-sig] Segmentation Fault with Boost.Python and
> Inheritance
> Message-ID: <61DAADFBCA6D4C468EF8216B42EBAE48 at gaberivescorbett.com (mailto:61DAADFBCA6D4C468EF8216B42EBAE48 at gaberivescorbett.com)>
> Content-Type: text/plain; charset="utf-8"
> 
> Hello, 
> 
> I'm getting a seg fault when trying to call a virtual method on a base class from python. Code is below:
> 
> #include <memory>
> 
> namespace boost { 
> template<class T> const T* get_pointer(const std::shared_ptr<T>& ptr) 
> {
> return ptr.get();
> }
> 
> template<class T> T* get_pointer(std::shared_ptr<T>& ptr)
> {
> return ptr.get();
> }
> }
> 
> #include <Python.h>
> #include <boost/python.hpp>
> 
> namespace bp = boost::python;
> 
> class MyBase
> {
> public:
> MyBase(){}
> virtual ~MyBase(){}
> 
> virtual void baseTest() { printf("base test\n"); }
> };
> 
> class MyDerived : public MyBase
> {
> public:
> MyDerived() {}
> virtual ~MyDerived(){}
> 
> void derivedTest() { printf("derived test\n"); }
> };
> 
> BOOST_PYTHON_MODULE(PythonTest)
> {
> bp::class_<MyBase, std::shared_ptr<MyBase>>("MyBase")
> .def("baseTest", &MyBase::baseTest);
> 
> bp::class_<MyDerived, bp::bases<MyBase>, std::shared_ptr<MyDerived>>("MyDerived")
> .def("derivedTest", &MyDerived::derivedTest);
> 
> bp::implicitly_convertible<std::shared_ptr<MyDerived>, std::shared_ptr<MyBase>>();
> }
> 
> Does it have to do with using std::shared_ptr for storage? If so is there a way around this? Any help is appreciated. 
> 
> -- 
> Gabe Rives-Corbett
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120517/c42e81a8/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 2
> Date: Fri, 18 May 2012 10:41:03 +0200
> From: Jonas Wielicki <j.wielicki at sotecware.net (mailto:j.wielicki at sotecware.net)>
> To: cplusplus-sig at python.org (mailto:cplusplus-sig at python.org)
> Subject: Re: [C++-sig] Segmentation Fault with Boost.Python and
> Inheritance
> Message-ID: <4FB60B1F.2050708 at sotecware.net (mailto:4FB60B1F.2050708 at sotecware.net)>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> On 17/05/12 23:42, Gabe Rives-Corbett wrote:
> > I'm getting a seg fault when trying to call a virtual method on a base class from python.
> 
> I had a similar problem a few days ago. Which g++ version are you using?
> I had problems with 4.7.0 (redhat) while 4.6.3 (redhat) was working fine.
> 
> -- 
> Jonas
> 
> 
> ------------------------------
> 
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org (mailto:Cplusplus-sig at python.org)
> http://mail.python.org/mailman/listinfo/cplusplus-sig
> 
> End of Cplusplus-sig Digest, Vol 44, Issue 17
> *********************************************
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120518/8149bf96/attachment-0001.html>

From j.wielicki at sotecware.net  Sat May 19 12:01:33 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Sat, 19 May 2012 12:01:33 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FB10488.3070802@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FB0E086.24110.64539BB9@s_sourceforge.nedprod.com>,
	<4FB0E42C.7050308@sotecware.net>
	<4FB0ED87.23804.64866704@s_sourceforge.nedprod.com>
	<4FB10488.3070802@sotecware.net>
Message-ID: <4FB76F7D.7000403@sotecware.net>

Okay, due to the other thread over there, I came back to this topic even
after I reverted to fedora 16 due to some other issues (they actually
removed ddate. Still cannot believe!)

I am working on a bugreport for the gcc folks right now, and during that
I came across the release notes for gcc 4.7[1] and found the following.

However, it seems that the actual problematic code is located in the
libboost itself. I am wondering how to report that properly. The gcc
folks want the -save-temps flag, but I really do not look through the
boost build system enough to know which file is to be built with the
correct option to get the result the gcc folks want. Any pointers here?

Its somewhere about inheritance.cpp:405.

-- 
Jonas

From j.wielicki at sotecware.net  Sat May 19 12:45:11 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Sat, 19 May 2012 12:45:11 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FB76F7D.7000403@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FB0E086.24110.64539BB9@s_sourceforge.nedprod.com>,
	<4FB0E42C.7050308@sotecware.net>
	<4FB0ED87.23804.64866704@s_sourceforge.nedprod.com>
	<4FB10488.3070802@sotecware.net> <4FB76F7D.7000403@sotecware.net>
Message-ID: <4FB779B7.6010704@sotecware.net>

On 19.05.2012 12:01, Jonas Wielicki wrote:
> I am working on a bugreport for the gcc folks right now, and during that
> I came across the release notes for gcc 4.7[1] and found the following.

missed reference, sorry:
[1] http://gcc.gnu.org/gcc-4.7/changes.html#cxx

-- 
Jonas

From j.wielicki at sotecware.net  Sun May 20 12:00:27 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Sun, 20 May 2012 12:00:27 +0200
Subject: [C++-sig] Segmentation Fault with Boost.Python and Inheritance
In-Reply-To: <82586BA9580946A79BAE019F64BBC4EE@gaberivescorbett.com>
References: <mailman.25.1337335202.6798.cplusplus-sig@python.org>
	<82586BA9580946A79BAE019F64BBC4EE@gaberivescorbett.com>
Message-ID: <4FB8C0BB.50309@sotecware.net>

On 19.05.2012 00:43, Gabe Rives-Corbett wrote:
> Is there any possible workaround to use inheritance with GCC 4.7? 
I know of none, sorry. I'd like to use more c++11 features too, but for
now I am forced to stick with 4.6.3. I am working on a bugreport on that
topic though. However, my time is pretty limited currently, so it might
take another few days until I get it submitted.

-- 
Jonas

From j.reid at mail.cryst.bbk.ac.uk  Mon May 21 10:44:47 2012
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Mon, 21 May 2012 09:44:47 +0100
Subject: [C++-sig] ANN: pyicl,
	a python interval container package based on boost.icl
Message-ID: <jpcva0$se$1@dough.gmane.org>

pyicl is a python package that exposes the functionality of boost.icl to
python using boost.python.

Documentation: http://packages.python.org/PyICL/
PyPi page: http://pypi.python.org/pypi/PyICL/

boost.icl is a general purpose interval container library written in
C++. Its author, Joachim Faulhaber, describes it thus:

"Intervals are almost ubiquitous in software development. Yet they are
very easily coded into user defined classes by a pair of numbers so they
are only implicitly used most of the time. The meaning of an interval is
simple. They represent all the elements between their lower and upper
bound and thus a set. But unlike sets, intervals usually can not be
added to a single new interval. If you want to add intervals to a
collection of intervals that does still represent a set, you arrive at
the idea of interval_sets provided by this library.

Interval containers of the ICL have been developed initially at Cortex
Software GmbH to solve problems related to date and time interval
computations in the context of a Hospital Information System. Time
intervals with associated values like amount of invoice or set of
therapies had to be manipulated in statistics, billing programs and
therapy scheduling programs. So the ICL emerged out of those industrial
use cases. It extracts generic code that helps to solve common problems
from the date and time problem domain and can be beneficial in other
fields as well.

One of the most advantageous aspects of interval containers is their
very compact representation of sets and maps. Working with sets and maps
of elements can be very inefficient, if in a given problem domain,
elements are typically occurring in contiguous chunks. Besides a compact
representation of associative containers, that can reduce the cost of
space and time drastically, the ICL comes with a universal mechanism of
aggregation, that allows to combine associated values in meaningful ways
when intervals overlap on insertion."

Regards,
John.


From s_sourceforge at nedprod.com  Mon May 21 17:55:53 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Mon, 21 May 2012 16:55:53 +0100
Subject: [C++-sig] boost::python with virtual inheritance and g++
	c++0x/11 (testcase attached)
In-Reply-To: <4FB76F7D.7000403@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>, <4FB10488.3070802@sotecware.net>,
	<4FB76F7D.7000403@sotecware.net>
Message-ID: <4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com>

On 19 May 2012 at 12:01, Jonas Wielicki wrote:

> I am working on a bugreport for the gcc folks right now, and during that
> I came across the release notes for gcc 4.7[1] and found the following.

That's a huge amount of new compliance with C++11. No wonder there 
are bugs.

> However, it seems that the actual problematic code is located in the
> libboost itself. I am wondering how to report that properly. The gcc
> folks want the -save-temps flag, but I really do not look through the
> boost build system enough to know which file is to be built with the
> correct option to get the result the gcc folks want. Any pointers here?
> 
> Its somewhere about inheritance.cpp:405.

1. Does the bug occur in non-optimised as well as optimised builds?

2. Does the bug occur when C++11 is turned off?

3. What are your -fvisibility settings?

4. Does the bug occur when inlining is force switched off?

5. Are you using precompiled headers? If so, does the bug occur when 
you turn those off?

6. Can you spot where in the assembly it's making a pointer 
dereferencing mistake?

I would be surprised if inheritance.cpp actually contains the 
offending code. More likely it's in the template headers, and 
therefore the compiler chooses one of the copies assembled in each 
compiland to go into the final binary. If that's the case, which file 
you choose for save temps is immaterial.

If you're *really* unlucky, the bug is that different assembler is 
being generated in different compilands and the fact you're seeing a 
problem is due to sheer chance because of the stochastic choice the 
linker made :)

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From j.wielicki at sotecware.net  Mon May 21 18:43:06 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Mon, 21 May 2012 18:43:06 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com>
References: <4FB0B8CC.3050001@sotecware.net>, <4FB10488.3070802@sotecware.net>,
	<4FB76F7D.7000403@sotecware.net>
	<4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com>
Message-ID: <4FBA709A.9020905@sotecware.net>

On 21.05.2012 17:55, Niall Douglas wrote:
> 1. Does the bug occur in non-optimised as well as optimised builds?
It does.

> 2. Does the bug occur when C++11 is turned off?
It does not.

> 3. What are your -fvisibility settings?
Unchanged to default... Does this help any? Can I find out whats default
on my machine if its configurable at gcc-build-time?

> 4. Does the bug occur when inlining is force switched off?
Yes.

> 5. Are you using precompiled headers? If so, does the bug occur when 
> you turn those off?
I think I am not. I would know if I did, wouldn't I? At least I can
browse through the header sources at /usr/include/boost/..

> 6. Can you spot where in the assembly it's making a pointer 
> dereferencing mistake?
That is in the template, but the pointer value changes inside code which
is compiled in the boost library.

Within the function defined around inheritance.cpp:392, the value of the
object pointer (I think it's called p) changes. I was yet unable to find
the specific point where the value is changed, because a lot of
subfunctions get called in there and, to be honest, I'm not that
familiar with gcc yet. Also it seems as maybe the wrong value is only
passed, while it is still intact on stack (gcc at least shows me
differing values for the two stackframes), but that might be due to
debug data or gcc magic?

> If you're *really* unlucky, the bug is that different assembler is 
> being generated in different compilands and the fact you're seeing a 
> problem is due to sheer chance because of the stochastic choice the 
> linker made :)
Actually, the problem is deterministic. I compiled the binary many times
now and with gcc 4.7 I always get the segfault, at the same instruction,
with the same surroundings (changed pointer value from one stack frame
to the other).

-- 
Jonas

From spartan811 at hotmail.com  Tue May 22 14:38:09 2012
From: spartan811 at hotmail.com (Yoann Chaumy)
Date: Tue, 22 May 2012 14:38:09 +0200
Subject: [C++-sig] function return a shared_ptr
In-Reply-To: <4FBA709A.9020905@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FB10488.3070802@sotecware.net>, , <4FB76F7D.7000403@sotecware.net>,
	<4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com>,
	<4FBA709A.9020905@sotecware.net>
Message-ID: <BLU158-W479F649F44A67A3A6B90DC9C020@phx.gbl>


 Hi, I will use a shared pointer to my Python script
I expose my class : 
class_<handler::Target, boost::shared_ptr<handler::Target> >("Target")
     .def("getRadius", &handler::Target::getRadius)
     .def("setRadius", &handler::Target::setRadius)
     .def("setPosition", &handler::Target::setPosition)
    ;  and I use a method "getTestTarget" who return a shared pointer on my member variable vgd::Shp<vgSofa::handler::Target> InterfaceScriptBoost::getTestTarget()
{
      return UlisBoost::m_testTarget;
}
  And the exposition of is methode is :  def("getTestTarget", &dtUlis::InterfaceScriptBoost::getTestTarget, python::return_value_policy<python::reference_existing_object>());
  I want to know if it is true ? Any help would be appreciated, thank you 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120522/06c756b2/attachment.html>

From s_sourceforge at nedprod.com  Tue May 22 16:43:06 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Tue, 22 May 2012 15:43:06 +0100
Subject: [C++-sig] boost::python with virtual inheritance and g++
	c++0x/11 (testcase attached)
In-Reply-To: <4FBA709A.9020905@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com>,
	<4FBA709A.9020905@sotecware.net>
Message-ID: <4FBBA5FA.16443.9488DA4@s_sourceforge.nedprod.com>

On 21 May 2012 at 18:43, Jonas Wielicki wrote:

> On 21.05.2012 17:55, Niall Douglas wrote:
> > 1. Does the bug occur in non-optimised as well as optimised builds?
> It does.

Joy.

> > 2. Does the bug occur when C++11 is turned off?
> It does not.

Not so much joy. It could, technically speaking, be a 
misinterpretation of C++03 by either BPL or GCC. Still, at least it 
narrows things down.

> > 5. Are you using precompiled headers? If so, does the bug occur when 
> > you turn those off?
> I think I am not. I would know if I did, wouldn't I? At least I can
> browse through the header sources at /usr/include/boost/..

Precompiled headers on GCC are basically a dump of state just after 
processing the headers. As a result, the file is huge. That might 
help you find out if they're on. I believe bjam defaults them to off.

> > 6. Can you spot where in the assembly it's making a pointer 
> > dereferencing mistake?
> That is in the template, but the pointer value changes inside code which
> is compiled in the boost library.

Are you saying that the *offset* changes inside the code?

So, class A has vtable at this[-8].

I access a->foo() which indexes this[-8] by 0x16 in the headers.

But inside the compiland, a->foo() indexes this[-8] by 0x20, which is 
wrong.

> Within the function defined around inheritance.cpp:392, the value of the
> object pointer (I think it's called p) changes. I was yet unable to find
> the specific point where the value is changed, because a lot of
> subfunctions get called in there and, to be honest, I'm not that
> familiar with gcc yet. Also it seems as maybe the wrong value is only
> passed, while it is still intact on stack (gcc at least shows me
> differing values for the two stackframes), but that might be due to
> debug data or gcc magic?

Yeah the GCC ABI is a bit fusty. Got a lot better in 3.2 onwards 
though. You might find 
http://sourcery.mentor.com/public/cxx-abi/cxx-vtable-ex.html useful 
as a reference for the C++03 ABI.

Of course, almost certainly the bug you're seeing is related to the 
ABI changes they're making for C++11, of which there are quite a few. 
If I had to take a guess, your problems might have something to do 
with fixing bugs in decltype support e.g. one of the changelog items 
in 4.7 is "The representation of C++ virtual thunks and aliases (both 
implicit and defined via the alias attribute) has been re-engineered. 
Aliases no longer pose optimization barriers and calls to an alias 
can be inlined and otherwise optimized."

BTW - can I just clarify you ARE compiling the entire of BPL using 
C++11 throughout? Linking C++11 to C++03 is *supposed* to work (but 
not the other way round), but I can see nests of vipers in it.

> > If you're *really* unlucky, the bug is that different assembler is 
> > being generated in different compilands and the fact you're seeing a 
> > problem is due to sheer chance because of the stochastic choice the 
> > linker made :)
> Actually, the problem is deterministic. I compiled the binary many times
> now and with gcc 4.7 I always get the segfault, at the same instruction,
> with the same surroundings (changed pointer value from one stack frame
> to the other).

Given the information so far, you have an excellent chance of getting 
it fixed.

Another thought - I once persuaded the GCC devs to accept a bug when 
I demonstrated that ICC (Intel's compiler) got it right. ICC is free 
on Linux, so that might be worth a shot too.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From talljimbo at gmail.com  Tue May 22 17:20:48 2012
From: talljimbo at gmail.com (Jim Bosch)
Date: Tue, 22 May 2012 11:20:48 -0400
Subject: [C++-sig] function return a shared_ptr
In-Reply-To: <BLU158-W479F649F44A67A3A6B90DC9C020@phx.gbl>
References: <4FB0B8CC.3050001@sotecware.net> <4FB10488.3070802@sotecware.net>
	<4FB76F7D.7000403@sotecware.net>
	<4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com>
	<4FBA709A.9020905@sotecware.net>
	<BLU158-W479F649F44A67A3A6B90DC9C020@phx.gbl>
Message-ID: <CAN+0q27nk7CNgFVYJw7minwNKGzV6gw_iYnL2m-nYKxXJz=WZw@mail.gmail.com>

On May 22, 2012 8:38 AM, "Yoann Chaumy" <spartan811 at hotmail.com> wrote:
>
>
> Hi,
> I will use a shared pointer to my Python script
>
> I expose my class :
>
> class_<handler::Target, boost::shared_ptr<handler::Target> >("Target")
>      .def("getRadius", &handler::Target::getRadius)
>      .def("setRadius", &handler::Target::setRadius)
>      .def("setPosition", &handler::Target::setPosition)
>     ;
>
>
> and I use a method "getTestTarget" who return a shared pointer on my
member variable
>
> vgd::Shp<vgSofa::handler::Target> InterfaceScriptBoost::getTestTarget()
> {
>       return UlisBoost::m_testTarget;
> }
>
>
>
> And the exposition of is methode is :
>
> def("getTestTarget", &dtUlis::InterfaceScriptBoost::getTestTarget,
python::return_value_policy<python::reference_existing_object>());
>
>
>
> I want to know if it is true ?
>
> Any help would be appreciated, thank you
>

If you are returning a smart pointer, there is no need to use a call policy
like return_value_policy<...>.  However, you have wrapped the class with
boost::shared_ptr and are returning a different kind of shared pointer.
That won't work.  It's easier if you can just use boost::shared_ptr
everywhere, but it is also possible to set up conversions for a custom
smart pointer.

HTH

Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120522/5bf561c9/attachment.html>

From gabe at gaberivescorbett.com  Tue May 22 18:07:28 2012
From: gabe at gaberivescorbett.com (Gabe Rives-Corbett)
Date: Tue, 22 May 2012 12:07:28 -0400
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FBBA5FA.16443.9488DA4@s_sourceforge.nedprod.com>
References: <4FB0B8CC.3050001@sotecware.net> <,>
	<4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com> <,>
	<4FBA709A.9020905@sotecware.net>
	<4FBBA5FA.16443.9488DA4@s_sourceforge.nedprod.com>
Message-ID: <7E262BE03A744729BC2A26DAE6A7E08F@gaberivescorbett.com>

When I had this problem I had not built BPL using C++11, how would I instruct bjam to use gcc at a specific location and with the -std=c++0x flag?

-- 
Gabe Rives-Corbett
Cell: (805) 570-8395


On Tuesday, May 22, 2012 at 10:43 AM, Niall Douglas wrote:

> On 21 May 2012 at 18:43, Jonas Wielicki wrote:
> 
> > On 21.05.2012 17:55, Niall Douglas wrote:
> > > 1. Does the bug occur in non-optimised as well as optimised builds?
> > 
> > It does.
> > 
> 
> 
> Joy.
> 
> > > 2. Does the bug occur when C++11 is turned off?
> > It does not.
> > 
> 
> 
> Not so much joy. It could, technically speaking, be a 
> misinterpretation of C++03 by either BPL or GCC. Still, at least it 
> narrows things down.
> 
> > > 5. Are you using precompiled headers? If so, does the bug occur when 
> > > you turn those off?
> > > 
> > 
> > I think I am not. I would know if I did, wouldn't I? At least I can
> > browse through the header sources at /usr/include/boost/..
> > 
> 
> 
> Precompiled headers on GCC are basically a dump of state just after 
> processing the headers. As a result, the file is huge. That might 
> help you find out if they're on. I believe bjam defaults them to off.
> 
> > > 6. Can you spot where in the assembly it's making a pointer 
> > > dereferencing mistake?
> > > 
> > 
> > That is in the template, but the pointer value changes inside code which
> > is compiled in the boost library.
> > 
> 
> 
> Are you saying that the *offset* changes inside the code?
> 
> So, class A has vtable at this[-8].
> 
> I access a->foo() which indexes this[-8] by 0x16 in the headers.
> 
> But inside the compiland, a->foo() indexes this[-8] by 0x20, which is 
> wrong.
> 
> > Within the function defined around inheritance.cpp:392, the value of the
> > object pointer (I think it's called p) changes. I was yet unable to find
> > the specific point where the value is changed, because a lot of
> > subfunctions get called in there and, to be honest, I'm not that
> > familiar with gcc yet. Also it seems as maybe the wrong value is only
> > passed, while it is still intact on stack (gcc at least shows me
> > differing values for the two stackframes), but that might be due to
> > debug data or gcc magic?
> > 
> 
> 
> Yeah the GCC ABI is a bit fusty. Got a lot better in 3.2 onwards 
> though. You might find 
> http://sourcery.mentor.com/public/cxx-abi/cxx-vtable-ex.html useful 
> as a reference for the C++03 ABI.
> 
> Of course, almost certainly the bug you're seeing is related to the 
> ABI changes they're making for C++11, of which there are quite a few. 
> If I had to take a guess, your problems might have something to do 
> with fixing bugs in decltype support e.g. one of the changelog items 
> in 4.7 is "The representation of C++ virtual thunks and aliases (both 
> implicit and defined via the alias attribute) has been re-engineered. 
> Aliases no longer pose optimization barriers and calls to an alias 
> can be inlined and otherwise optimized."
> 
> BTW - can I just clarify you ARE compiling the entire of BPL using 
> C++11 throughout? Linking C++11 to C++03 is *supposed* to work (but 
> not the other way round), but I can see nests of vipers in it.
> 
> > > If you're *really* unlucky, the bug is that different assembler is 
> > > being generated in different compilands and the fact you're seeing a 
> > > problem is due to sheer chance because of the stochastic choice the 
> > > linker made :)
> > > 
> > 
> > Actually, the problem is deterministic. I compiled the binary many times
> > now and with gcc 4.7 I always get the segfault, at the same instruction,
> > with the same surroundings (changed pointer value from one stack frame
> > to the other).
> > 
> 
> 
> Given the information so far, you have an excellent chance of getting 
> it fixed.
> 
> Another thought - I once persuaded the GCC devs to accept a bug when 
> I demonstrated that ICC (Intel's compiler) got it right. ICC is free 
> on Linux, so that might be worth a shot too.
> 
> Niall
> 
> -- 
> Technology & Consulting Services - ned Productions Limited.
> http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
> Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
> 
> 
> 
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org (mailto:Cplusplus-sig at python.org)
> http://mail.python.org/mailman/listinfo/cplusplus-sig
> 
> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120522/751b892d/attachment-0001.html>

From j.wielicki at sotecware.net  Tue May 22 18:33:11 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Tue, 22 May 2012 18:33:11 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FBBA5FA.16443.9488DA4@s_sourceforge.nedprod.com>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com>,
	<4FBA709A.9020905@sotecware.net>
	<4FBBA5FA.16443.9488DA4@s_sourceforge.nedprod.com>
Message-ID: <4FBBBFC7.3040902@sotecware.net>

On 22.05.2012 16:43, Niall Douglas wrote:
> Precompiled headers on GCC are basically a dump of state just after 
> processing the headers. As a result, the file is huge. That might 
> help you find out if they're on. I believe bjam defaults them to off.
Well, in that case I'm not using them. They would've sprung into my eyes
if they're located in the default -I-paths (which I assume).

>>> 6. Can you spot where in the assembly it's making a pointer 
>>> dereferencing mistake?
>> That is in the template, but the pointer value changes inside code which
>> is compiled in the boost library.
> 
> Are you saying that the *offset* changes inside the code?
Hm, I was unclear I guess. I'm going to paste some gdb bt output:

#0  0x0000000000405af7 in
boost::python::objects::polymorphic_id_generator<Foo>::execute
(p_=0x40a268) at /usr/include/boost/python/object/inheritance.hpp:43
#1  0x00007ffff7dcb684 in boost::(anonymous namespace)::convert_type (p=
    0x666db0, src_t=..., dst_t=..., polymorphic=true)
    at
/usr/src/debug/boost_1_48_0/libs/python/src/object/inheritance.cpp:405

Here we see that to convert_type, p=0x666db0 has been passed, while
0x40a268 arrived in execute. While the first value is a valid pointer to
a Foo instance, 0x40a268 is anything but that (actually I did not find
out yet what was going on there).
However, p is just passed on to execute without modification (as far as
I can see in the code).

> BTW - can I just clarify you ARE compiling the entire of BPL using 
> C++11 throughout? Linking C++11 to C++03 is *supposed* to work (but 
> not the other way round), but I can see nests of vipers in it.
Ehm, I guess not, so I just downloaded and rebuilt boost 1.48 in the
virtual machine (no fun) with -std=c++11 flag (I added this in the
project-options.jam file at the using line; running ./b2 -d2 showed me
that the flag was passed to the gcc too). Still, the crash occurs at the
same place.

> Given the information so far, you have an excellent chance of getting 
> it fixed.
Okay, so I'm taking you suggest I submit both compilation attempts
(inheritance.cpp and crashtest.cpp) with -save-temps for comparision to
the gcc devs? Of course with all the information I gathered so far.

-- 
Jonas


From j.wielicki at sotecware.net  Tue May 22 18:35:33 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Tue, 22 May 2012 18:35:33 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <7E262BE03A744729BC2A26DAE6A7E08F@gaberivescorbett.com>
References: <4FB0B8CC.3050001@sotecware.net> <,
	> <4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com> <,
	> <4FBA709A.9020905@sotecware.net>
	<4FBBA5FA.16443.9488DA4@s_sourceforge.nedprod.com>
	<7E262BE03A744729BC2A26DAE6A7E08F@gaberivescorbett.com>
Message-ID: <4FBBC055.404@sotecware.net>

On 22.05.2012 18:07, Gabe Rives-Corbett wrote:
> how would I instruct bjam to use gcc at a specific location and with the -std=c++0x flag?

I found that the following works if you replace the existing section of
the project-config.jam file in the boost source tarball:

>>> snip
# Compiler configuration. This definition will be used unless
# you already have defined some toolsets in your user-config.jam
# file.
if ! gcc in [ feature.values <toolset> ]
{
    using gcc : : : <compileflags>-std=c++11 ;
}
<<< snip

Verify that it works for you by calling b2 or bjam with -d2. This will
print you the g++ calls to the terminal. Check that the -std=c++11 flag
is present there.

-- 
Jonas

From s_sourceforge at nedprod.com  Tue May 22 19:05:57 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Tue, 22 May 2012 18:05:57 +0100
Subject: [C++-sig] boost::python with virtual inheritance and g++
	c++0x/11 (testcase attached)
In-Reply-To: <4FBBBFC7.3040902@sotecware.net>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FBBA5FA.16443.9488DA4@s_sourceforge.nedprod.com>,
	<4FBBBFC7.3040902@sotecware.net>
Message-ID: <4FBBC775.9056.9CB559F@s_sourceforge.nedprod.com>

On 22 May 2012 at 18:33, Jonas Wielicki wrote:

> > BTW - can I just clarify you ARE compiling the entire of BPL using 
> > C++11 throughout? Linking C++11 to C++03 is *supposed* to work (but 
> > not the other way round), but I can see nests of vipers in it.
> Ehm, I guess not, so I just downloaded and rebuilt boost 1.48 in the
> virtual machine (no fun) with -std=c++11 flag (I added this in the
> project-options.jam file at the using line; running ./b2 -d2 showed me
> that the flag was passed to the gcc too). Still, the crash occurs at the
> same place.

Ok.

> > Given the information so far, you have an excellent chance of getting 
> > it fixed.
> Okay, so I'm taking you suggest I submit both compilation attempts
> (inheritance.cpp and crashtest.cpp) with -save-temps for comparision to
> the gcc devs? Of course with all the information I gathered so far.

Include the answers to the questions I asked about force inlining, is 
BPL compiled as c++11, the problem doesn't show in c++03 etc. It 
shows you've been thorough.

Do post the bugzilla link here, I'd like to CC into it.

Thanks,
Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From j.wielicki at sotecware.net  Tue May 22 19:43:10 2012
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Tue, 22 May 2012 19:43:10 +0200
Subject: [C++-sig] boost::python with virtual inheritance and g++
 c++0x/11 (testcase attached)
In-Reply-To: <4FBBC775.9056.9CB559F@s_sourceforge.nedprod.com>
References: <4FB0B8CC.3050001@sotecware.net>,
	<4FBBA5FA.16443.9488DA4@s_sourceforge.nedprod.com>,
	<4FBBBFC7.3040902@sotecware.net>
	<4FBBC775.9056.9CB559F@s_sourceforge.nedprod.com>
Message-ID: <4FBBD02E.9030902@sotecware.net>

On 22.05.2012 19:05, Niall Douglas wrote:
> Do post the bugzilla link here, I'd like to CC into it.
There you go:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53455

I hope they won't murder me for the .gz ;).

-- 
Jonas

From spartan811 at hotmail.com  Wed May 23 09:36:06 2012
From: spartan811 at hotmail.com (Yoann Chaumy)
Date: Wed, 23 May 2012 09:36:06 +0200
Subject: [C++-sig] function return a shared_ptr
In-Reply-To: <CAN+0q27nk7CNgFVYJw7minwNKGzV6gw_iYnL2m-nYKxXJz=WZw@mail.gmail.com>
References: <4FB0B8CC.3050001@sotecware.net>
	<4FB10488.3070802@sotecware.net>, <4FB76F7D.7000403@sotecware.net>,
	<4FBA6589.3571.464D5B3@s_sourceforge.nedprod.com>,
	<4FBA709A.9020905@sotecware.net>,
	<BLU158-W479F649F44A67A3A6B90DC9C020@phx.gbl>,
	<CAN+0q27nk7CNgFVYJw7minwNKGzV6gw_iYnL2m-nYKxXJz=WZw@mail.gmail.com>
Message-ID: <BLU158-W55C01F0061A248A7A6DB8C9C030@phx.gbl>



 >If you are returning a smart pointer, there is no need to use a call policy like return_value_policy<...>.  However, you have wrapped the class with boost::shared_ptr and are returning a different kind of shared pointer.  That won't work.  It's easier if you can just use boost::shared_ptr everywhere, but it is also possible to set up conversions for a custom smart pointer.  Ok thank you.  But I have a error, when I try to expose my class :  Assertion failed!
Program: ...
File: libs\python\src\converter\registry.cpp
Line: 207
Expression: slot->m_to_python == 0  I am using MS Visual Express 2010 

 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120523/01cf3210/attachment.html>

From sybren at stuvel.eu  Wed May 23 16:21:42 2012
From: sybren at stuvel.eu (=?UTF-8?Q?Sybren_A=2E_St=C3=BCvel?=)
Date: Wed, 23 May 2012 16:21:42 +0200
Subject: [C++-sig] Boost::Python: overriding and smart pointers
Message-ID: <CAAAfPCAurtabSpmwYhSCgvjDbL5rT231MRuaGy+AgLTtrV_LMQ@mail.gmail.com>

Dear list,

I'm trying to get my head around shared pointers combined with overloaded
methods. I want to create instances of my class both in Python and C++, but
for some reason I can't get both to work with the same code. I've
constructed a minimal example to show you what I'm trying to do. The full
C++ code with syntax highlighting can be found at
http://pastebin.com/ZqG79QUL

Sorry for the long mail, but I saw no other way to show my issues in a
clear and unambiguous way.

--------------------------------------
class ExampleObject {
public:
    std::string name;

    ExampleObject(const std::string & name) : name(name) {}
    virtual ~ExampleObject() {}
    virtual int some_number() { return 12; }
};

typedef boost::shared_ptr<ExampleObject> ExampleObjectPtr;

ExampleObjectPtr create_example_object(const std::string & name) {
    return boost::make_shared<ExampleObject>(name);
}

void print_name(ExampleObjectPtr object) {
    std::cout << "Example object named '" << object->name
        << "', nr = " << object->some_number()
        << std::endl;
}
--------------------------------------

These are the class and functions that I want to wrap. The
ExampleObject::some_number method should be overridable in Python. To this
end, I've created the following wrapper class:

--------------------------------------
struct ExampleObject_wrapper : ExampleObject, wrapper<ExampleObject> {
    ExampleObject_wrapper(const std::string & name)
        : ExampleObject(name), wrapper<ExampleObject>()
    {}

    virtual int some_number(void) override {
        if( override func_override = this->get_override("some_number"))
            return func_override();
        return default_some_number();
    }

    int default_some_number(void) {
        return this->ExampleObject::some_number();
    }
};

typedef boost::shared_ptr<ExampleObject_wrapper> ExampleObject_wrapper_ptr;
--------------------------------------

The Python module is declared as follows:

--------------------------------------
BOOST_PYTHON_MODULE(wraptest) {
    def("create_example_object", &create_example_object);
    def("print_name", &print_name);

    class_<ExampleObject_wrapper, boost::noncopyable, ExampleObjectPtr>
        ("ExampleObject", init<const std::string &>())
        .def("some_number", &ExampleObject_wrapper::some_number,
                            &ExampleObject_wrapper::default_some_number)
    ;
}
--------------------------------------

This is the Python code that I'm testing with:

--------------------------------------
from wraptest import *

class ExampleSubclass(ExampleObject):
    def some_number(self):
        return ExampleObject.some_number(self) * 2

# Test from Python
from_py_obj = ExampleSubclass('from python')
print_name(from_py_obj)

# Test from C++
from_cpp = create_example_object('from C++')
print_name(from_cpp)
--------------------------------------

When I compile the module (VS2010, Python 3.2.2, Boost 1.48) and run the
Python script, I would expect the object created in C++ to show the number
12, and the object created in Python to show the number 24. However, the
output is this:

Example object named 'from python', nr = 12
Example object named 'from C++', nr = 12

Running a debugger also shows that the ExampleObject_wrapper methods aren't
called at all. When I edit the class_<...> line by replacing
ExampleObjectPtr with ExampleObject_wrapper_ptr, the from-Python object
works just fine, but the from-C++ object gives me an exception:

Example object named 'from python', nr = 24
Traceback (most recent call last):
  File "C:\workspace\rage\scripts\test_wrapping.py", line 13, in <module>
    from_cpp = create_example_object('from C++')
TypeError: No to_python (by-value) converter found for C++ type: class
boost::shared_ptr<class ExampleObject>

What should I do to be able to instantiate objects in C++ and Python alike?

Kind regards,
-- 
Sybren A. St?vel

http://stuvel.eu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120523/4f03f995/attachment.html>

From ndbecker2 at gmail.com  Thu May 24 15:50:34 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Thu, 24 May 2012 09:50:34 -0400
Subject: [C++-sig] Segmentation Fault with Boost.Python and Inheritance
References: <61DAADFBCA6D4C468EF8216B42EBAE48@gaberivescorbett.com>
	<4FB68B42.31250.7A79E9DE@s_sourceforge.nedprod.com>
	<4FB694C0.1050306@sotecware.net>
	<4FB696D4.5445.7AA71B63@s_sourceforge.nedprod.com>
Message-ID: <jpleba$ler$1@dough.gmane.org>

I believe I hit the same bug, using extract on a polymorphic type.

In the old code, I used make_constructor, passing object to it, and using 
extract.

In the new code, I don't use make_constructor, just use bp::init.  My segfault 
is gone.  Maybe just luck?


From adam.preble at gmail.com  Sat May 26 20:19:12 2012
From: adam.preble at gmail.com (Adam Preble)
Date: Sat, 26 May 2012 13:19:12 -0500
Subject: [C++-sig] Mysterious triggerings of "UNREF invalid object" in
	_Py_ForgetReference
Message-ID: <CAE-m0_mDj77p-Md4xDE+ebT0yb0Mu0PH7NGThU-z2rF3udaYBA@mail.gmail.com>

This might be one for the main Python lists but since I have a whole lot of
stuff wrapped in Boost floating about this problem, I wanted to try the
C++-sig for starters.  I run my little game experiment for anywhere between
15 and 60 seconds, where it's sending a lot of events and messages around
between my C++ runtime and the Python runtime.  The code that's failing
often completes hundreds of times without fault.  I don't know when this
started to happen, but it's something of a recent phenomenon.  Given the
asynchronous stuff in my code, this could have been latent in it the whole
time.

I'm specifically using Stackless Python 2.6.5 with Boost 1.49, with debug
symbols, so I'll paste _Py_ForgetReference so it's in front of you:

void
_Py_ForgetReference(register PyObject *op)
{
#ifdef SLOW_UNREF_CHECK
        register PyObject *p;
#endif
if (op->ob_refcnt < 0)
Py_FatalError("UNREF negative refcnt");
if (op == &refchain ||
    op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
Py_FatalError("UNREF invalid object");
#ifdef SLOW_UNREF_CHECK
for (p = refchain._ob_next; p != &refchain; p = p->_ob_next) {
if (p == op)
break;
}
if (p == &refchain) /* Not found */
Py_FatalError("UNREF unknown object");
#endif
op->_ob_next->_ob_prev = op->_ob_prev;
op->_ob_prev->_ob_next = op->_ob_next;
op->_ob_next = op->_ob_prev = NULL;
_Py_INC_TPFREES(op);
}

Here's where I get bit:

if (op == &refchain ||
    op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
Py_FatalError("UNREF invalid object");

I am developing in Visual Studio 2010, and I use the immediate window to
test those logic clauses.
There are two general situations where it happens:

1. A shared pointer to a message I created in the C++ runtime was passed to
an object in the Python runtime, processed, and the control was returned
back.  It secured and released the GIL in and out of that Python crossing.
 On the way out of the original C++ function it naturally decrements the
shared_ptr use count and starts to destroy it.  That is what I want to
happen.  Call stack of relevant bits:

  python26_d.dll!Py_FatalError(const char * msg)  Line 1679 C
python26_d.dll!_Py_ForgetReference(_object * op)  Line 2178 + 0xa bytes C
  python26_d.dll!_Py_Dealloc(_object * op)  Line 2197 + 0x9 bytes C
  wva.exe!boost::python::xdecref<_object>(_object * p)  Line 36 + 0xb3 bytes
C++
  wva.exe!boost::python::handle<_object>::reset()  Line 249 + 0xb bytes C++
  wva.exe!boost::python::converter::shared_ptr_deleter::operator()(const
void * __formal)  Line 36 C++
  wva.exe!boost::detail::sp_counted_impl_pd<void
*,boost::python::converter::shared_ptr_deleter>::dispose()  Line 149 C++
  wva.exe!boost::detail::sp_counted_base::release()  Line 102 + 0xf bytes
C++
  wva.exe!boost::detail::shared_count::~shared_count()  Line 309 C++

If I probe that if condition I see this:
op == &refchain
0
op->_ob_prev->_ob_next != op
0
op->_ob_next->_ob_prev != op
0

Nothing was true!  How could that conditional trigger?  All I can suppose
is a gremlin came in and changed a condition on me.  Something that has
concerned me is I don't grab the GIL when I deallocate these objects.  I
don't know how I'd do that.  I have suspected that was a liability for
awhile, but I'm not entirely sure how.

2. Within the same block of C++ code, at the point that I'm trying to
transmit the message to the Python-derived object, it'll puke too.  So here
it has already created a shared_ptr for the message and is triggering a
callback into the Python code to handle it.  The Python-derived object is
being called through a wrapper, and that call has the GIL.  This stack
trace is much more obnoxious and it's difficult for me to make any sense of
it.  Note there's some Stackless stuff in there.  I think the interpreter
is at least starting to call some of the code in the Python derivation, but
I haven't been able to figure out how far it gets.  I'll take any advice on
how to probe this stuff since I feel I am too vague here--look for square
brackets on a few lines for some things I figured out:

  python26_d.dll!Py_FatalError(const char * msg)  Line 1679 C
python26_d.dll!_Py_ForgetReference(_object * op)  Line 2178 C
  python26_d.dll!_Py_Dealloc(_object * op)  Line 2197 + 0x9 bytes C
  python26_d.dll!tupledealloc(PyTupleObject * op)  Line 170 + 0x86 bytes C
  python26_d.dll!_Py_Dealloc(_object * op)  Line 2198 + 0x7 bytes C
  python26_d.dll!PyObject_CallFunctionObjArgs(_object * callable, ...)
 Line 2751 + 0x54 bytes C
  python26_d.dll!handle_callback(_PyWeakReference * ref, _object *
callback)  Line 881 + 0xf bytes C
  python26_d.dll!PyObject_ClearWeakRefs(_object * object)  Line 928 + 0xd
bytes C
  wva.exe!instance_dealloc(_object * inst)  Line 344 + 0xa bytes C++
 [This is Boost.Python class.cpp, statically linked]
  python26_d.dll!subtype_dealloc(_object * self)  Line 1020 + 0x7 bytes C
  python26_d.dll!_Py_Dealloc(_object * op)  Line 2198 + 0x7 bytes C
       [I know here it's deallocating a wrapped type for a 3d vector I was
passing around]
  python26_d.dll!insertdict(_dictobject * mp, _object * key, long hash,
_object * value)  Line 459 + 0x54 bytes C      [It has replaced an existing
3d vector with the passed one, and trying to nuke the old one]
  python26_d.dll!PyDict_SetItem(_object * op, _object * key, _object *
value)  Line 701 + 0x15 bytes C
  python26_d.dll!PyObject_GenericSetAttr(_object * obj, _object * name,
_object * value)  Line 1504 + 0x11 bytes C
  python26_d.dll!PyObject_SetAttr(_object * v, _object * name, _object *
value)  Line 1247 + 0x14 bytes C      [value is my 3d vector I am passing
around]
  python26_d.dll!PyEval_EvalFrame_value(_frame * f, int throwflag, _object
* retval)  Line 2063 C
  python26_d.dll!PyEval_EvalFrameEx_slp(_frame * f, int throwflag, _object
* retval)  Line 836 + 0x15 bytes C
  python26_d.dll!slp_frame_dispatch_top(_object * retval)  Line 719 + 0x12
bytes C
  python26_d.dll!slp_run_tasklet()  Line 1204 + 0x9 bytes C
  python26_d.dll!slp_eval_frame(_frame * f)  Line 299 + 0x5 bytes C
  python26_d.dll!climb_stack_and_eval_frame(_frame * f)  Line 266 + 0x9
bytes C
  python26_d.dll!slp_eval_frame(_frame * f)  Line 294 + 0x9 bytes C
  python26_d.dll!PyEval_EvalCodeEx(PyCodeObject * co, _object * globals,
_object * locals, _object * * args, int argcount, _object * * kws, int
kwcount, _object * * defs, int defcount, _object * closure)  Line 3294 +
0x6 bytes C
  python26_d.dll!function_call(_object * func, _object * arg, _object * kw)
 Line 540 + 0x40 bytes C
  python26_d.dll!PyObject_Call(_object * func, _object * arg, _object * kw)
 Line 2502 + 0x3c bytes C
  python26_d.dll!instancemethod_call(_object * func, _object * arg, _object
* kw)  Line 2586 + 0x11 bytes C
  python26_d.dll!PyObject_Call(_object * func, _object * arg, _object * kw)
 Line 2502 + 0x3c bytes C
  python26_d.dll!PyEval_CallObjectWithKeywords(_object * func, _object *
arg, _object * kw)  Line 3931 + 0x11 bytes C
  python26_d.dll!PyEval_CallFunction(_object * obj, const char * format,
...)  Line 556 + 0xf bytes C
  wva.exe!boost::python::override::operator()<boost::shared_ptr<game::IComponentCommunicatable>,unsigned
int,boost::shared_ptr<game::ComponentMessage> >(const
boost::shared_ptr<game::IComponentCommunicatable> & a0, const unsigned int
& a1, const boost::shared_ptr<game::ComponentMessage> & a2)  Line 138 +
0xac bytes C++
  wva.exe!game::ComponentWrapper::IncomingSignalEvent(boost::shared_ptr<game::IComponentCommunicatable>
source, unsigned int id, boost::shared_ptr<game::ComponentMessage> message)
 Line 234 + 0x4b bytes C++

At least this time the logic condition is true...
op == &refchain
0
op->_ob_prev->_ob_next != op
1
op->_ob_next->_ob_prev != op
0

What it's trying to free is of type __PyWeakref_RefType.

The 3d vector wrapper has a lot of methods, but I think of particular
interest would be the class block.  It looks like
this: class_<vector3df>("vector3df", init<float, float, float>())
Stuff in Python can create them and they can also get passed around from
C++ to Python and back.

The impression I get is that something is getting freed before its time.  I
couldn't tell if it's the shared_ptr self-destructing or the Python GC
jumping the gun.

I don't have a good impression is what is wrong so I'm finding it hard to
write a simplified, self-contained example for the list.  I didn't expect a
silver bullet with this first message, but I figured somebody had enough
experience that I could start isolating things and pare it down.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120526/648e2d59/attachment-0001.html>

From s_sourceforge at nedprod.com  Sun May 27 16:05:44 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Sun, 27 May 2012 15:05:44 +0100
Subject: [C++-sig] Mysterious triggerings of "UNREF invalid object" in
	_Py_ForgetReference
In-Reply-To: <CAE-m0_mDj77p-Md4xDE+ebT0yb0Mu0PH7NGThU-z2rF3udaYBA@mail.gmail.com>
References: <CAE-m0_mDj77p-Md4xDE+ebT0yb0Mu0PH7NGThU-z2rF3udaYBA@mail.gmail.com>
Message-ID: <4FC234B8.30231.22E623F4@s_sourceforge.nedprod.com>

Try pinning everything to a single CPU, see what happens.

Try pinning the CPU clock speed to its minimum. If it doesn't trip, 
you have a timing race.

If you can build on Linux, try valgrind.

Oh, you mentioned bits you weren't locking when destroying. I'd lock 
those and see what happens.

Niall

On 26 May 2012 at 13:19, Adam Preble wrote:

> This might be one for the main Python lists but since I have a whole lot of
> stuff wrapped in Boost floating about this problem, I wanted to try the
> C++-sig for starters.  I run my little game experiment for anywhere between
> 15 and 60 seconds, where it's sending a lot of events and messages around
> between my C++ runtime and the Python runtime.  The code that's failing
> often completes hundreds of times without fault.  I don't know when this
> started to happen, but it's something of a recent phenomenon.  Given the
> asynchronous stuff in my code, this could have been latent in it the whole
> time.
> 
> I'm specifically using Stackless Python 2.6.5 with Boost 1.49, with debug
> symbols, so I'll paste _Py_ForgetReference so it's in front of you:
> 
> void
> _Py_ForgetReference(register PyObject *op)
> {
> #ifdef SLOW_UNREF_CHECK
>         register PyObject *p;
> #endif
> if (op->ob_refcnt < 0)
> Py_FatalError("UNREF negative refcnt");
> if (op == &refchain ||
>     op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
> Py_FatalError("UNREF invalid object");
> #ifdef SLOW_UNREF_CHECK
> for (p = refchain._ob_next; p != &refchain; p = p->_ob_next) {
> if (p == op)
> break;
> }
> if (p == &refchain) /* Not found */
> Py_FatalError("UNREF unknown object");
> #endif
> op->_ob_next->_ob_prev = op->_ob_prev;
> op->_ob_prev->_ob_next = op->_ob_next;
> op->_ob_next = op->_ob_prev = NULL;
> _Py_INC_TPFREES(op);
> }
> 
> Here's where I get bit:
> 
> if (op == &refchain ||
>     op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
> Py_FatalError("UNREF invalid object");
> 
> I am developing in Visual Studio 2010, and I use the immediate window to
> test those logic clauses.
> There are two general situations where it happens:
> 
> 1. A shared pointer to a message I created in the C++ runtime was passed to
> an object in the Python runtime, processed, and the control was returned
> back.  It secured and released the GIL in and out of that Python crossing.
>  On the way out of the original C++ function it naturally decrements the
> shared_ptr use count and starts to destroy it.  That is what I want to
> happen.  Call stack of relevant bits:
> 
>   python26_d.dll!Py_FatalError(const char * msg)  Line 1679 C
> python26_d.dll!_Py_ForgetReference(_object * op)  Line 2178 + 0xa bytes C
>   python26_d.dll!_Py_Dealloc(_object * op)  Line 2197 + 0x9 bytes C
>   wva.exe!boost::python::xdecref<_object>(_object * p)  Line 36 + 0xb3 bytes
> C++
>   wva.exe!boost::python::handle<_object>::reset()  Line 249 + 0xb bytes C++
>   wva.exe!boost::python::converter::shared_ptr_deleter::operator()(const
> void * __formal)  Line 36 C++
>   wva.exe!boost::detail::sp_counted_impl_pd<void
> *,boost::python::converter::shared_ptr_deleter>::dispose()  Line 149 C++
>   wva.exe!boost::detail::sp_counted_base::release()  Line 102 + 0xf bytes
> C++
>   wva.exe!boost::detail::shared_count::~shared_count()  Line 309 C++
> 
> If I probe that if condition I see this:
> op == &refchain
> 0
> op->_ob_prev->_ob_next != op
> 0
> op->_ob_next->_ob_prev != op
> 0
> 
> Nothing was true!  How could that conditional trigger?  All I can suppose
> is a gremlin came in and changed a condition on me.  Something that has
> concerned me is I don't grab the GIL when I deallocate these objects.  I
> don't know how I'd do that.  I have suspected that was a liability for
> awhile, but I'm not entirely sure how.
> 
> 2. Within the same block of C++ code, at the point that I'm trying to
> transmit the message to the Python-derived object, it'll puke too.  So here
> it has already created a shared_ptr for the message and is triggering a
> callback into the Python code to handle it.  The Python-derived object is
> being called through a wrapper, and that call has the GIL.  This stack
> trace is much more obnoxious and it's difficult for me to make any sense of
> it.  Note there's some Stackless stuff in there.  I think the interpreter
> is at least starting to call some of the code in the Python derivation, but
> I haven't been able to figure out how far it gets.  I'll take any advice on
> how to probe this stuff since I feel I am too vague here--look for square
> brackets on a few lines for some things I figured out:
> 
>   python26_d.dll!Py_FatalError(const char * msg)  Line 1679 C
> python26_d.dll!_Py_ForgetReference(_object * op)  Line 2178 C
>   python26_d.dll!_Py_Dealloc(_object * op)  Line 2197 + 0x9 bytes C
>   python26_d.dll!tupledealloc(PyTupleObject * op)  Line 170 + 0x86 bytes C
>   python26_d.dll!_Py_Dealloc(_object * op)  Line 2198 + 0x7 bytes C
>   python26_d.dll!PyObject_CallFunctionObjArgs(_object * callable, ...)
>  Line 2751 + 0x54 bytes C
>   python26_d.dll!handle_callback(_PyWeakReference * ref, _object *
> callback)  Line 881 + 0xf bytes C
>   python26_d.dll!PyObject_ClearWeakRefs(_object * object)  Line 928 + 0xd
> bytes C
>   wva.exe!instance_dealloc(_object * inst)  Line 344 + 0xa bytes C++
>  [This is Boost.Python class.cpp, statically linked]
>   python26_d.dll!subtype_dealloc(_object * self)  Line 1020 + 0x7 bytes C
>   python26_d.dll!_Py_Dealloc(_object * op)  Line 2198 + 0x7 bytes C
>        [I know here it's deallocating a wrapped type for a 3d vector I was
> passing around]
>   python26_d.dll!insertdict(_dictobject * mp, _object * key, long hash,
> _object * value)  Line 459 + 0x54 bytes C      [It has replaced an existing
> 3d vector with the passed one, and trying to nuke the old one]
>   python26_d.dll!PyDict_SetItem(_object * op, _object * key, _object *
> value)  Line 701 + 0x15 bytes C
>   python26_d.dll!PyObject_GenericSetAttr(_object * obj, _object * name,
> _object * value)  Line 1504 + 0x11 bytes C
>   python26_d.dll!PyObject_SetAttr(_object * v, _object * name, _object *
> value)  Line 1247 + 0x14 bytes C      [value is my 3d vector I am passing
> around]
>   python26_d.dll!PyEval_EvalFrame_value(_frame * f, int throwflag, _object
> * retval)  Line 2063 C
>   python26_d.dll!PyEval_EvalFrameEx_slp(_frame * f, int throwflag, _object
> * retval)  Line 836 + 0x15 bytes C
>   python26_d.dll!slp_frame_dispatch_top(_object * retval)  Line 719 + 0x12
> bytes C
>   python26_d.dll!slp_run_tasklet()  Line 1204 + 0x9 bytes C
>   python26_d.dll!slp_eval_frame(_frame * f)  Line 299 + 0x5 bytes C
>   python26_d.dll!climb_stack_and_eval_frame(_frame * f)  Line 266 + 0x9
> bytes C
>   python26_d.dll!slp_eval_frame(_frame * f)  Line 294 + 0x9 bytes C
>   python26_d.dll!PyEval_EvalCodeEx(PyCodeObject * co, _object * globals,
> _object * locals, _object * * args, int argcount, _object * * kws, int
> kwcount, _object * * defs, int defcount, _object * closure)  Line 3294 +
> 0x6 bytes C
>   python26_d.dll!function_call(_object * func, _object * arg, _object * kw)
>  Line 540 + 0x40 bytes C
>   python26_d.dll!PyObject_Call(_object * func, _object * arg, _object * kw)
>  Line 2502 + 0x3c bytes C
>   python26_d.dll!instancemethod_call(_object * func, _object * arg, _object
> * kw)  Line 2586 + 0x11 bytes C
>   python26_d.dll!PyObject_Call(_object * func, _object * arg, _object * kw)
>  Line 2502 + 0x3c bytes C
>   python26_d.dll!PyEval_CallObjectWithKeywords(_object * func, _object *
> arg, _object * kw)  Line 3931 + 0x11 bytes C
>   python26_d.dll!PyEval_CallFunction(_object * obj, const char * format,
> ...)  Line 556 + 0xf bytes C
>   wva.exe!boost::python::override::operator()<boost::shared_ptr<game::IComponentCommunicatable>,unsigned
> int,boost::shared_ptr<game::ComponentMessage> >(const
> boost::shared_ptr<game::IComponentCommunicatable> & a0, const unsigned int
> & a1, const boost::shared_ptr<game::ComponentMessage> & a2)  Line 138 +
> 0xac bytes C++
>   wva.exe!game::ComponentWrapper::IncomingSignalEvent(boost::shared_ptr<game::IComponentCommunicatable>
> source, unsigned int id, boost::shared_ptr<game::ComponentMessage> message)
>  Line 234 + 0x4b bytes C++
> 
> At least this time the logic condition is true...
> op == &refchain
> 0
> op->_ob_prev->_ob_next != op
> 1
> op->_ob_next->_ob_prev != op
> 0
> 
> What it's trying to free is of type __PyWeakref_RefType.
> 
> The 3d vector wrapper has a lot of methods, but I think of particular
> interest would be the class block.  It looks like
> this: class_<vector3df>("vector3df", init<float, float, float>())
> Stuff in Python can create them and they can also get passed around from
> C++ to Python and back.
> 
> The impression I get is that something is getting freed before its time.  I
> couldn't tell if it's the shared_ptr self-destructing or the Python GC
> jumping the gun.
> 
> I don't have a good impression is what is wrong so I'm finding it hard to
> write a simplified, self-contained example for the list.  I didn't expect a
> silver bullet with this first message, but I figured somebody had enough
> experience that I could start isolating things and pare it down.
> 


-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From adam.preble at gmail.com  Sun May 27 17:26:46 2012
From: adam.preble at gmail.com (Adam Preble)
Date: Sun, 27 May 2012 10:26:46 -0500
Subject: [C++-sig] Mysterious triggerings of "UNREF invalid object" in
	_Py_ForgetReference
In-Reply-To: <4FC234B8.30231.22E623F4@s_sourceforge.nedprod.com>
References: <CAE-m0_mDj77p-Md4xDE+ebT0yb0Mu0PH7NGThU-z2rF3udaYBA@mail.gmail.com>
	<4FC234B8.30231.22E623F4@s_sourceforge.nedprod.com>
Message-ID: <CAE-m0_=zrGfNz20L-K5_T-eZqFrXOZjMdmucjic85AEHGoc44Q@mail.gmail.com>

I don't know if I can build on Linux but I have see ways to play with the
clock and setting affinity.  However, I don't know how to take the GIL
properly during destruction.  What we're talking about is a shared_ptr to
an interface that was created on the Python side.  I can slap GIL
acquisition in the destructor but I know I'm not wrapping all the logic
that goes on to the Python runtime that way for freeing the object.  Do you
know how that would be done?

In the meanwhile I'm looking at the Stackless examples that relate to how
CCP games said they did their engine.  If I can make it work, it would mean
all Python stuff gets scheduled on one thread.  I am not entirely sure
about object destruction though.

On Sun, May 27, 2012 at 9:05 AM, Niall Douglas <s_sourceforge at nedprod.com>wrote:

> Try pinning everything to a single CPU, see what happens.
>
> Try pinning the CPU clock speed to its minimum. If it doesn't trip,
> you have a timing race.
>
> If you can build on Linux, try valgrind.
>
> Oh, you mentioned bits you weren't locking when destroying. I'd lock
> those and see what happens.
>
> Niall
>
> On 26 May 2012 at 13:19, Adam Preble wrote:
>
> > This might be one for the main Python lists but since I have a whole lot
> of
> > stuff wrapped in Boost floating about this problem, I wanted to try the
> > C++-sig for starters.  I run my little game experiment for anywhere
> between
> > 15 and 60 seconds, where it's sending a lot of events and messages around
> > between my C++ runtime and the Python runtime.  The code that's failing
> > often completes hundreds of times without fault.  I don't know when this
> > started to happen, but it's something of a recent phenomenon.  Given the
> > asynchronous stuff in my code, this could have been latent in it the
> whole
> > time.
> >
> > I'm specifically using Stackless Python 2.6.5 with Boost 1.49, with debug
> > symbols, so I'll paste _Py_ForgetReference so it's in front of you:
> >
> > void
> > _Py_ForgetReference(register PyObject *op)
> > {
> > #ifdef SLOW_UNREF_CHECK
> >         register PyObject *p;
> > #endif
> > if (op->ob_refcnt < 0)
> > Py_FatalError("UNREF negative refcnt");
> > if (op == &refchain ||
> >     op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
> > Py_FatalError("UNREF invalid object");
> > #ifdef SLOW_UNREF_CHECK
> > for (p = refchain._ob_next; p != &refchain; p = p->_ob_next) {
> > if (p == op)
> > break;
> > }
> > if (p == &refchain) /* Not found */
> > Py_FatalError("UNREF unknown object");
> > #endif
> > op->_ob_next->_ob_prev = op->_ob_prev;
> > op->_ob_prev->_ob_next = op->_ob_next;
> > op->_ob_next = op->_ob_prev = NULL;
> > _Py_INC_TPFREES(op);
> > }
> >
> > Here's where I get bit:
> >
> > if (op == &refchain ||
> >     op->_ob_prev->_ob_next != op || op->_ob_next->_ob_prev != op)
> > Py_FatalError("UNREF invalid object");
> >
> > I am developing in Visual Studio 2010, and I use the immediate window to
> > test those logic clauses.
> > There are two general situations where it happens:
> >
> > 1. A shared pointer to a message I created in the C++ runtime was passed
> to
> > an object in the Python runtime, processed, and the control was returned
> > back.  It secured and released the GIL in and out of that Python
> crossing.
> >  On the way out of the original C++ function it naturally decrements the
> > shared_ptr use count and starts to destroy it.  That is what I want to
> > happen.  Call stack of relevant bits:
> >
> >   python26_d.dll!Py_FatalError(const char * msg)  Line 1679 C
> > python26_d.dll!_Py_ForgetReference(_object * op)  Line 2178 + 0xa bytes C
> >   python26_d.dll!_Py_Dealloc(_object * op)  Line 2197 + 0x9 bytes C
> >   wva.exe!boost::python::xdecref<_object>(_object * p)  Line 36 + 0xb3
> bytes
> > C++
> >   wva.exe!boost::python::handle<_object>::reset()  Line 249 + 0xb bytes
> C++
> >   wva.exe!boost::python::converter::shared_ptr_deleter::operator()(const
> > void * __formal)  Line 36 C++
> >   wva.exe!boost::detail::sp_counted_impl_pd<void
> > *,boost::python::converter::shared_ptr_deleter>::dispose()  Line 149 C++
> >   wva.exe!boost::detail::sp_counted_base::release()  Line 102 + 0xf bytes
> > C++
> >   wva.exe!boost::detail::shared_count::~shared_count()  Line 309 C++
> >
> > If I probe that if condition I see this:
> > op == &refchain
> > 0
> > op->_ob_prev->_ob_next != op
> > 0
> > op->_ob_next->_ob_prev != op
> > 0
> >
> > Nothing was true!  How could that conditional trigger?  All I can suppose
> > is a gremlin came in and changed a condition on me.  Something that has
> > concerned me is I don't grab the GIL when I deallocate these objects.  I
> > don't know how I'd do that.  I have suspected that was a liability for
> > awhile, but I'm not entirely sure how.
> >
> > 2. Within the same block of C++ code, at the point that I'm trying to
> > transmit the message to the Python-derived object, it'll puke too.  So
> here
> > it has already created a shared_ptr for the message and is triggering a
> > callback into the Python code to handle it.  The Python-derived object is
> > being called through a wrapper, and that call has the GIL.  This stack
> > trace is much more obnoxious and it's difficult for me to make any sense
> of
> > it.  Note there's some Stackless stuff in there.  I think the interpreter
> > is at least starting to call some of the code in the Python derivation,
> but
> > I haven't been able to figure out how far it gets.  I'll take any advice
> on
> > how to probe this stuff since I feel I am too vague here--look for square
> > brackets on a few lines for some things I figured out:
> >
> >   python26_d.dll!Py_FatalError(const char * msg)  Line 1679 C
> > python26_d.dll!_Py_ForgetReference(_object * op)  Line 2178 C
> >   python26_d.dll!_Py_Dealloc(_object * op)  Line 2197 + 0x9 bytes C
> >   python26_d.dll!tupledealloc(PyTupleObject * op)  Line 170 + 0x86 bytes
> C
> >   python26_d.dll!_Py_Dealloc(_object * op)  Line 2198 + 0x7 bytes C
> >   python26_d.dll!PyObject_CallFunctionObjArgs(_object * callable, ...)
> >  Line 2751 + 0x54 bytes C
> >   python26_d.dll!handle_callback(_PyWeakReference * ref, _object *
> > callback)  Line 881 + 0xf bytes C
> >   python26_d.dll!PyObject_ClearWeakRefs(_object * object)  Line 928 + 0xd
> > bytes C
> >   wva.exe!instance_dealloc(_object * inst)  Line 344 + 0xa bytes C++
> >  [This is Boost.Python class.cpp, statically linked]
> >   python26_d.dll!subtype_dealloc(_object * self)  Line 1020 + 0x7 bytes C
> >   python26_d.dll!_Py_Dealloc(_object * op)  Line 2198 + 0x7 bytes C
> >        [I know here it's deallocating a wrapped type for a 3d vector I
> was
> > passing around]
> >   python26_d.dll!insertdict(_dictobject * mp, _object * key, long hash,
> > _object * value)  Line 459 + 0x54 bytes C      [It has replaced an
> existing
> > 3d vector with the passed one, and trying to nuke the old one]
> >   python26_d.dll!PyDict_SetItem(_object * op, _object * key, _object *
> > value)  Line 701 + 0x15 bytes C
> >   python26_d.dll!PyObject_GenericSetAttr(_object * obj, _object * name,
> > _object * value)  Line 1504 + 0x11 bytes C
> >   python26_d.dll!PyObject_SetAttr(_object * v, _object * name, _object *
> > value)  Line 1247 + 0x14 bytes C      [value is my 3d vector I am passing
> > around]
> >   python26_d.dll!PyEval_EvalFrame_value(_frame * f, int throwflag,
> _object
> > * retval)  Line 2063 C
> >   python26_d.dll!PyEval_EvalFrameEx_slp(_frame * f, int throwflag,
> _object
> > * retval)  Line 836 + 0x15 bytes C
> >   python26_d.dll!slp_frame_dispatch_top(_object * retval)  Line 719 +
> 0x12
> > bytes C
> >   python26_d.dll!slp_run_tasklet()  Line 1204 + 0x9 bytes C
> >   python26_d.dll!slp_eval_frame(_frame * f)  Line 299 + 0x5 bytes C
> >   python26_d.dll!climb_stack_and_eval_frame(_frame * f)  Line 266 + 0x9
> > bytes C
> >   python26_d.dll!slp_eval_frame(_frame * f)  Line 294 + 0x9 bytes C
> >   python26_d.dll!PyEval_EvalCodeEx(PyCodeObject * co, _object * globals,
> > _object * locals, _object * * args, int argcount, _object * * kws, int
> > kwcount, _object * * defs, int defcount, _object * closure)  Line 3294 +
> > 0x6 bytes C
> >   python26_d.dll!function_call(_object * func, _object * arg, _object *
> kw)
> >  Line 540 + 0x40 bytes C
> >   python26_d.dll!PyObject_Call(_object * func, _object * arg, _object *
> kw)
> >  Line 2502 + 0x3c bytes C
> >   python26_d.dll!instancemethod_call(_object * func, _object * arg,
> _object
> > * kw)  Line 2586 + 0x11 bytes C
> >   python26_d.dll!PyObject_Call(_object * func, _object * arg, _object *
> kw)
> >  Line 2502 + 0x3c bytes C
> >   python26_d.dll!PyEval_CallObjectWithKeywords(_object * func, _object *
> > arg, _object * kw)  Line 3931 + 0x11 bytes C
> >   python26_d.dll!PyEval_CallFunction(_object * obj, const char * format,
> > ...)  Line 556 + 0xf bytes C
> >
> wva.exe!boost::python::override::operator()<boost::shared_ptr<game::IComponentCommunicatable>,unsigned
> > int,boost::shared_ptr<game::ComponentMessage> >(const
> > boost::shared_ptr<game::IComponentCommunicatable> & a0, const unsigned
> int
> > & a1, const boost::shared_ptr<game::ComponentMessage> & a2)  Line 138 +
> > 0xac bytes C++
> >
> wva.exe!game::ComponentWrapper::IncomingSignalEvent(boost::shared_ptr<game::IComponentCommunicatable>
> > source, unsigned int id, boost::shared_ptr<game::ComponentMessage>
> message)
> >  Line 234 + 0x4b bytes C++
> >
> > At least this time the logic condition is true...
> > op == &refchain
> > 0
> > op->_ob_prev->_ob_next != op
> > 1
> > op->_ob_next->_ob_prev != op
> > 0
> >
> > What it's trying to free is of type __PyWeakref_RefType.
> >
> > The 3d vector wrapper has a lot of methods, but I think of particular
> > interest would be the class block.  It looks like
> > this: class_<vector3df>("vector3df", init<float, float, float>())
> > Stuff in Python can create them and they can also get passed around from
> > C++ to Python and back.
> >
> > The impression I get is that something is getting freed before its time.
>  I
> > couldn't tell if it's the shared_ptr self-destructing or the Python GC
> > jumping the gun.
> >
> > I don't have a good impression is what is wrong so I'm finding it hard to
> > write a simplified, self-contained example for the list.  I didn't
> expect a
> > silver bullet with this first message, but I figured somebody had enough
> > experience that I could start isolating things and pare it down.
> >
>
>
> --
> Technology & Consulting Services - ned Productions Limited.
> http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
> Work Portfolio: http://careers.stackoverflow.com/nialldouglas/
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120527/839eb103/attachment-0001.html>

From sureshj at gmail.com  Mon May 28 05:37:00 2012
From: sureshj at gmail.com (Suresh Joshi)
Date: Sun, 27 May 2012 20:37:00 -0700 (PDT)
Subject: [C++-sig] Injecting void* and HWND handling code from Py++
Message-ID: <1338176220019-4630570.post@n4.nabble.com>

Hello all!

I had a question about how to handle void* and HWNDs from within Python
using Boost::Python and Py++. Reading through the Py++ documentation, I'm
loving how I can slice and dice my classes/wrappers and seemingly do
whatever I want with them, but there's still A LOT I haven't figured out.

I've found a previous set of similar questions here:
http://boost.2283326.n4.nabble.com/Wrapping-and-Passing-HWND-with-Boost-Python-td4598953.html
but didn't fully understand the automated implementation details.

All the relevant code is posted at the end, but my main questions were as
follows.

(1) Using the win32gui Python library, HWNDs are passed around as ints
describing the address of the window. These can be reinterpret_cast'ed into
HWNDs in C++ and that seems to work fine. However, using Py++, how can I
automatically inject a reinterpret_cast into Foo's constructor, or the
initializer list of a constructor wrapper? 

I've found these commands,
.add_declaration_code(...)
.add_registration_code(...)
.add_wrapper_code(...)
but they don't place code where I need it to go. 

So, is there a command that I should be using? My goal is to not touch my
Foo.hpp code (as in, no added code, no helper code), so I would like
everything to stay within the Boost::Python generated wrapper code
(FooWrapper.cpp).

(2) From Python, I would like to manipulate the memory of a buffer stored
within Foo. In my REAL code, the only accessor I can use is Address() which
returns a void pointer, which Py++/BP flips into an opaque pointer (which I
don't think I can use to manipulate m_buffer's memory in Python). 

One option is ctypes, but I frankly just don't know how to combine that with
BP for the purpose of manipulating data behind a void*, even after having
looked at the ctypes integration code in the Py++ docs.

The other option is to do something like what I did with the fake VoidPtr()
accessor. I could create accessors like VoidPtrAsUChar() for example, and
then expose those using the return_addressof call policy (hopefully!). If
this is, in fact, a valid approach, then I have the same question as
above... 

Given that I can't change the Foo.hpp code, is there a clean way of
injecting those AsUChar, AsInt, As... helper methods straight into the
wrapper code, so that they call Address() internally and then
reinterpret_cast the returned void* into something more useful and
accessible in Python?

Thanks!
-SJ

--------------------------------------------------
Here is my trivial Foo class:

#include <windows.h>
class Foo
{
public: 
    Foo( const HWND window )
        : m_wnd( window )
    {
        const unsigned int size = 10;
        m_buffer = new unsigned char[size];
        for( unsigned int i = 0; i < size; ++i )
            m_buffer[i] = i+1;
    }

    ~Foo()
    {   delete[] m_buffer;  }

    int IsWindow()
    {
        if ( ::IsWindow( m_wnd ) )
            return 1;
        return -1;
    }

    void* Address() // In my real code, this is the interface to access
memory
    { return reinterpret_cast< void* >( m_buffer ); }

    // Fake example accessors
    void* VoidPtr()
    { return reinterpret_cast< void* >( m_buffer ); }

    int* VoidPtrAsInt() 
    { return reinterpret_cast< int* >( m_buffer ); } 

    unsigned char* VoidPtrAsUChar() 
    { return reinterpret_cast< unsigned char* >( m_buffer ); } 

    char* VoidPtrAsChar() 
    { return reinterpret_cast< char* >( m_buffer ); } 

private:
    HWND m_wnd;
    unsigned char* m_buffer; // This buffer could be of any type in real
code

    Foo();
    Foo(const Foo&);
};

--------------------------------------------------
Here is my Py++ code:

mb = module_builder.module_builder_t(   files=fileList  
                                      , gccxml_path=pathToGccXml
                                      , working_directory=workingDirectory 
                                      , compiler='msvc9' )
                                      

mb.member_functions( return_type='char *' ).call_policies =
call_policies.return_value_policy( call_policies.return_addressof )
mb.member_functions( return_type='int *' ).call_policies =
call_policies.return_value_policy( call_policies.return_addressof )
mb.member_functions( return_type='unsigned char *' ).call_policies =
call_policies.return_value_policy( call_policies.return_addressof )

mb.build_code_creator( module_name='FooWrapper' )
FooWrapper = os.path.join( os.path.abspath('.'), 'FooWrapper.cpp' )

if os.path.exists( FooWrapper ):
    os.remove( FooWrapper )

mb.write_module( FooWrapper )

--------------------------------------------------
Which produces this generated code:
namespace bp = boost::python;

BOOST_PYTHON_MODULE(FooWrapper){
    { //::Foo
        typedef bp::class_< Foo, boost::noncopyable > Foo_exposer_t;
        Foo_exposer_t Foo_exposer = Foo_exposer_t( "Foo", bp::init< HWND__ *
>(( bp::arg("window") )) );
        bp::scope Foo_scope( Foo_exposer );
        bp::implicitly_convertible< HWND const, Foo >();
        { //::Foo::Address
        
            typedef void * ( ::Foo::*Address_function_type )(  ) ;
            
            Foo_exposer.def( 
                "Address"
                , Address_function_type( &::Foo::Address )
                , bp::return_value_policy< bp::return_opaque_pointer >() );
        
        }
        { //::Foo::IsWindow
        
            typedef int ( ::Foo::*IsWindow_function_type )(  ) ;
            
            Foo_exposer.def( 
                "IsWindow"
                , IsWindow_function_type( &::Foo::IsWindow ) );
        
        }
        { //::Foo::VoidPtr
        
            typedef void * ( ::Foo::*VoidPtr_function_type )(  ) ;
            
            Foo_exposer.def( 
                "VoidPtr"
                , VoidPtr_function_type( &::Foo::VoidPtr )
                , bp::return_value_policy< bp::return_opaque_pointer >() );
        
        }
        { //::Foo::VoidPtrAsChar
        
            typedef char * ( ::Foo::*VoidPtrAsChar_function_type )(  ) ;
            
            Foo_exposer.def( 
                "VoidPtrAsChar"
                , VoidPtrAsChar_function_type( &::Foo::VoidPtrAsChar )
                , bp::return_value_policy<
pyplusplus::call_policies::return_addressof >() );
        
        }
        { //::Foo::VoidPtrAsInt
        
            typedef int * ( ::Foo::*VoidPtrAsInt_function_type )(  ) ;
            
            Foo_exposer.def( 
                "VoidPtrAsInt"
                , VoidPtrAsInt_function_type( &::Foo::VoidPtrAsInt )
                , bp::return_value_policy<
pyplusplus::call_policies::return_addressof >() );
        
        }
        { //::Foo::VoidPtrAsUChar
        
            typedef unsigned char * ( ::Foo::*VoidPtrAsUChar_function_type
)(  ) ;
            
            Foo_exposer.def( 
                "VoidPtrAsUChar"
                , VoidPtrAsUChar_function_type( &::Foo::VoidPtrAsUChar )
                , bp::return_value_policy<
pyplusplus::call_policies::return_addressof >() );
        
        }
    }
}

--
View this message in context: http://boost.2283326.n4.nabble.com/Injecting-void-and-HWND-handling-code-from-Py-tp4630570.html
Sent from the Python - c++-sig mailing list archive at Nabble.com.

From adam.preble at gmail.com  Tue May 29 04:39:30 2012
From: adam.preble at gmail.com (Adam Preble)
Date: Mon, 28 May 2012 21:39:30 -0500
Subject: [C++-sig] Injecting instance of object into main module
Message-ID: <CAE-m0_kcSJ7Qw=G1e13mxk56f1wpGXqwTzXL1oZRfki6GtaKXQ@mail.gmail.com>

I wanted to provide an instance of a particular object I'm using for
Stackless Python state management.  I'm trying to take over the green
thread management so I can do my own scheduling--but that's only context.
 In the attached code, I'm trying to do some of life's more simple things.
 I have the main module, and the scope for that module.  With this, I had
hoped I could embed an actual instance of an object.  From there I
anticipate calling it from Python without having to import or do anything
else.  I figured I'd take the main script scope, add an object called
"runtime", and set it to the one in my C++ runtime.

It compiles fine but blows up during runtime with the ever-descriptive
"error_already_set()" without a message.

This problem only seems to touch the Boost part of the runtime so I don't
think that's an issue.  I'm using Stackless Python 2.6.5, with Boost 1.49,
on Windows Vista with Visual Studio 2010.

I had as my inspiration this:
http://wiki.python.org/moin/boost.python/HowTo#module_level_objects

Which doesn't look to be current.  I can't figure out how to use the
run-time example there, in particular.  It looks like calling interpreter()
is not supported, or I have no idea how to make it happen.

I'm doing something like this:
EmbeddedSynchronization embeddedSync;        // A class member up above
...

[Did the PyInitialize]
boost::python::handle<>
mainHandle(boost::python::borrowed(PyImport_AddModule("__main__")));
boost::python::object   mainModule(mainHandle);
boost::python::scope    mainScope(mainModule);

boost::python::object   class_oClassSync =
boost::python::class_<EmbeddedSynchronization,
boost::noncopyable>("EmbeddedSynchronization")
                                            .def("nap",
&EmbeddedSynchronization::beNice)
                                            .staticmethod("nap");

mainScope().attr("runtime") = embeddedSync;         // Dead right here

Stack trace going down in there:
> StacklessEmbedding.exe!boost::python::throw_error_already_set()  Line 62
C++
  StacklessEmbedding.exe!boost::python::expect_non_null<_object>(_object *
x)  Line 46 C++
  StacklessEmbedding.exe!boost::python::converter::detail::return_object_manager_from_python<boost::python::api::object>::operator()(_object
* obj)  Line 156 + 0x9 bytes C++
  StacklessEmbedding.exe!boost::python::call<boost::python::api::object>(_object
* callable, boost::type<boost::python::api::object> * __formal)  Line 76 +
0x10 bytes C++
  StacklessEmbedding.exe!boost::python::api::object_operators<boost::python::api::object>::operator()()
 Line 54 + 0x14 bytes C++

Anything obviously wrong?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120528/3a9a1c94/attachment.html>

From roman.yakovenko at gmail.com  Tue May 29 15:33:26 2012
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Tue, 29 May 2012 16:33:26 +0300
Subject: [C++-sig] Injecting void* and HWND handling code from Py++
In-Reply-To: <1338176220019-4630570.post@n4.nabble.com>
References: <1338176220019-4630570.post@n4.nabble.com>
Message-ID: <CABYYnMz7m2zEG0f5GvHgg5w67dY6NFd4sEHD=uYZ89sj=xAW8w@mail.gmail.com>

On Mon, May 28, 2012 at 6:37 AM, Suresh Joshi <sureshj at gmail.com> wrote:
> Hello all!
>
> I had a question about how to handle void* and HWNDs from within Python
> using Boost::Python and Py++. Reading through the Py++ documentation, I'm
> loving how I can slice and dice my classes/wrappers and seemingly do
> whatever I want with them, but there's still A LOT I haven't figured out.
>
> I've found a previous set of similar questions here:
> http://boost.2283326.n4.nabble.com/Wrapping-and-Passing-HWND-with-Boost-Python-td4598953.html
> but didn't fully understand the automated implementation details.
>
> All the relevant code is posted at the end, but my main questions were as
> follows.
>
> (1) Using the win32gui Python library, HWNDs are passed around as ints
> describing the address of the window. These can be reinterpret_cast'ed into
> HWNDs in C++ and that seems to work fine. However, using Py++, how can I
> automatically inject a reinterpret_cast into Foo's constructor, or the
> initializer list of a constructor wrapper?
>
> I've found these commands,
> .add_declaration_code(...)
> .add_registration_code(...)
> .add_wrapper_code(...)
> but they don't place code where I need it to go.
>
> So, is there a command that I should be using? My goal is to not touch my
> Foo.hpp code (as in, no added code, no helper code), so I would like
> everything to stay within the Boost::Python generated wrapper code
> (FooWrapper.cpp).

What you can do is to generate global function, which takes HWND as
integer, does casting and returns your object (via std::auto_ptr).
Then, you can add registration code, that use
boost::python::make_constructor function to create your class.

> (2) From Python, I would like to manipulate the memory of a buffer stored
> within Foo. In my REAL code, the only accessor I can use is Address() which
> returns a void pointer, which Py++/BP flips into an opaque pointer (which I
> don't think I can use to manipulate m_buffer's memory in Python).
>
> One option is ctypes, but I frankly just don't know how to combine that with
> BP for the purpose of manipulating data behind a void*, even after having
> looked at the ctypes integration code in the Py++ docs.
>
> The other option is to do something like what I did with the fake VoidPtr()
> accessor. I could create accessors like VoidPtrAsUChar() for example, and
> then expose those using the return_addressof call policy (hopefully!). If
> this is, in fact, a valid approach, then I have the same question as
> above...
>
> Given that I can't change the Foo.hpp code, is there a clean way of
> injecting those AsUChar, AsInt, As... helper methods straight into the
> wrapper code, so that they call Address() internally and then
> reinterpret_cast the returned void* into something more useful and
> accessible in Python?

I think Py++ repository contains ctypes + boost.python integration
code example. It worse to check it.

Regards,
Roman.

From s_sourceforge at nedprod.com  Tue May 29 18:32:18 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Tue, 29 May 2012 17:32:18 +0100
Subject: [C++-sig] Mysterious triggerings of "UNREF invalid object" in
	_Py_ForgetReference
In-Reply-To: <CAE-m0_=zrGfNz20L-K5_T-eZqFrXOZjMdmucjic85AEHGoc44Q@mail.gmail.com>
References: <CAE-m0_mDj77p-Md4xDE+ebT0yb0Mu0PH7NGThU-z2rF3udaYBA@mail.gmail.com>,
	<4FC234B8.30231.22E623F4@s_sourceforge.nedprod.com>,
	<CAE-m0_=zrGfNz20L-K5_T-eZqFrXOZjMdmucjic85AEHGoc44Q@mail.gmail.com>
Message-ID: <4FC4FA12.28027.2DB94040@s_sourceforge.nedprod.com>

On 27 May 2012 at 10:26, Adam Preble wrote:

> I don't know if I can build on Linux but I have see ways to play with the
> clock and setting affinity.  However, I don't know how to take the GIL
> properly during destruction.  What we're talking about is a shared_ptr to
> an interface that was created on the Python side.  I can slap GIL
> acquisition in the destructor but I know I'm not wrapping all the logic
> that goes on to the Python runtime that way for freeing the object.  Do you
> know how that would be done?

Sure. Get the smart pointer to context the GIL for you. It's what I 
do in my code. I have a policy constructed smart pointer, so it's 
fairly straightforward to have it call a magic virtual function just 
before destruction. That basically upcalls down the inheritance tree 
to prepare the object for destruction, effectively zombiefying it.

> In the meanwhile I'm looking at the Stackless examples that relate to how
> CCP games said they did their engine.  If I can make it work, it would mean
> all Python stuff gets scheduled on one thread.  I am not entirely sure
> about object destruction though.

Generally it's easiest to keep python on a single thread, except as a 
method to do async i/o. All that reference counting is deadly across 
CPU cores for cache coherency.

Increasingly actually I'm finding myself not using threads in my new 
code. SMP has hard scaling limits, and threads only work well inside 
a SMP node, not across them. I'm finding myself using more and more 
batch processing in my new code, it scales wonderfully.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From s_sourceforge at nedprod.com  Tue May 29 19:03:58 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Tue, 29 May 2012 18:03:58 +0100
Subject: [C++-sig] Injecting instance of object into main module
In-Reply-To: <CAE-m0_kcSJ7Qw=G1e13mxk56f1wpGXqwTzXL1oZRfki6GtaKXQ@mail.gmail.com>
References: <CAE-m0_kcSJ7Qw=G1e13mxk56f1wpGXqwTzXL1oZRfki6GtaKXQ@mail.gmail.com>
Message-ID: <4FC5017E.21579.2DD63F77@s_sourceforge.nedprod.com>

You're passing a null object pointer, so BPL correctly fails an 
assertion check?

Niall

On 28 May 2012 at 21:39, Adam Preble wrote:

> I wanted to provide an instance of a particular object I'm using for
> Stackless Python state management.  I'm trying to take over the green
> thread management so I can do my own scheduling--but that's only context.
>  In the attached code, I'm trying to do some of life's more simple things.
>  I have the main module, and the scope for that module.  With this, I had
> hoped I could embed an actual instance of an object.  From there I
> anticipate calling it from Python without having to import or do anything
> else.  I figured I'd take the main script scope, add an object called
> "runtime", and set it to the one in my C++ runtime.
> 
> It compiles fine but blows up during runtime with the ever-descriptive
> "error_already_set()" without a message.
> 
> This problem only seems to touch the Boost part of the runtime so I don't
> think that's an issue.  I'm using Stackless Python 2.6.5, with Boost 1.49,
> on Windows Vista with Visual Studio 2010.
> 
> I had as my inspiration this:
> http://wiki.python.org/moin/boost.python/HowTo#module_level_objects
> 
> Which doesn't look to be current.  I can't figure out how to use the
> run-time example there, in particular.  It looks like calling interpreter()
> is not supported, or I have no idea how to make it happen.
> 
> I'm doing something like this:
> EmbeddedSynchronization embeddedSync;        // A class member up above
> ...
> 
> [Did the PyInitialize]
> boost::python::handle<>
> mainHandle(boost::python::borrowed(PyImport_AddModule("__main__")));
> boost::python::object   mainModule(mainHandle);
> boost::python::scope    mainScope(mainModule);
> 
> boost::python::object   class_oClassSync =
> boost::python::class_<EmbeddedSynchronization,
> boost::noncopyable>("EmbeddedSynchronization")
>                                             .def("nap",
> &EmbeddedSynchronization::beNice)
>                                             .staticmethod("nap");
> 
> mainScope().attr("runtime") = embeddedSync;         // Dead right here
> 
> Stack trace going down in there:
> > StacklessEmbedding.exe!boost::python::throw_error_already_set()  Line 62
> C++
>   StacklessEmbedding.exe!boost::python::expect_non_null<_object>(_object *
> x)  Line 46 C++
>   StacklessEmbedding.exe!boost::python::converter::detail::return_object_manager_from_python<boost::python::api::object>::operator()(_object
> * obj)  Line 156 + 0x9 bytes C++
>   StacklessEmbedding.exe!boost::python::call<boost::python::api::object>(_object
> * callable, boost::type<boost::python::api::object> * __formal)  Line 76 +
> 0x10 bytes C++
>   StacklessEmbedding.exe!boost::python::api::object_operators<boost::python::api::object>::operator()()
>  Line 54 + 0x14 bytes C++
> 
> Anything obviously wrong?
> 


-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From s_sourceforge at nedprod.com  Tue May 29 20:23:03 2012
From: s_sourceforge at nedprod.com (Niall Douglas)
Date: Tue, 29 May 2012 19:23:03 +0100
Subject: [C++-sig] Injecting void* and HWND handling code from Py++
In-Reply-To: <1338176220019-4630570.post@n4.nabble.com>
References: <1338176220019-4630570.post@n4.nabble.com>
Message-ID: <4FC51407.30098.2E1EA6CE@s_sourceforge.nedprod.com>

On 27 May 2012 at 20:37, Suresh Joshi wrote:

> (2) From Python, I would like to manipulate the memory of a buffer stored
> within Foo. In my REAL code, the only accessor I can use is Address() which
> returns a void pointer, which Py++/BP flips into an opaque pointer (which I
> don't think I can use to manipulate m_buffer's memory in Python). 
> 
> One option is ctypes, but I frankly just don't know how to combine that with
> BP for the purpose of manipulating data behind a void*, even after having
> looked at the ctypes integration code in the Py++ docs.
> 
> The other option is to do something like what I did with the fake VoidPtr()
> accessor. I could create accessors like VoidPtrAsUChar() for example, and
> then expose those using the return_addressof call policy (hopefully!). If
> this is, in fact, a valid approach, then I have the same question as
> above... 
> 
> Given that I can't change the Foo.hpp code, is there a clean way of
> injecting those AsUChar, AsInt, As... helper methods straight into the
> wrapper code, so that they call Address() internally and then
> reinterpret_cast the returned void* into something more useful and
> accessible in Python?

If you really, really want to do this, I would have the wrapper of 
your Address method return an array-like interface e.g. like a 
std::vector<>. Look into the indexing suite - a method of 
implementing direct image bitmap access from python came up on this 
list a few years back.

Niall

-- 
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/




From adam.preble at gmail.com  Wed May 30 04:30:06 2012
From: adam.preble at gmail.com (Adam Preble)
Date: Tue, 29 May 2012 21:30:06 -0500
Subject: [C++-sig] Injecting instance of object into main module
In-Reply-To: <4FC5017E.21579.2DD63F77@s_sourceforge.nedprod.com>
References: <CAE-m0_kcSJ7Qw=G1e13mxk56f1wpGXqwTzXL1oZRfki6GtaKXQ@mail.gmail.com>
	<4FC5017E.21579.2DD63F77@s_sourceforge.nedprod.com>
Message-ID: <CAE-m0_k3y39wxTw1_nf=CHisJ3-X4O1UwXUyGTLT-iVRJctMqA@mail.gmail.com>

The object was declared in the class and instantiated when an instance of
that object is created.  It wasn't a pointer I failed to construct.  I
managed to screw around with the Python documentation awhile and get
something like this working:

    boost::python::handle<>
mainHandle(boost::python::borrowed(PyImport_AddModule("__main__")));
    boost::python::object   mainModule(mainHandle);
    boost::python::scope    mainScope(mainModule);

    boost::python::object main_namespace = mainModule.attr("__dict__");
    boost::python::object scripting_module(
(handle<>(PyImport_ImportModule("scripting_synchronization"))) );
    mainModule.attr("runtime") = ptr(&embeddedSync);

So it popped up in my main namespace as an object named "runtime."

On Tue, May 29, 2012 at 12:03 PM, Niall Douglas
<s_sourceforge at nedprod.com>wrote:

> You're passing a null object pointer, so BPL correctly fails an
> assertion check?
>
> Niall
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20120529/6a667ed4/attachment-0001.html>