From charlessolar at gmail.com  Mon Feb  1 16:22:09 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Mon, 1 Feb 2010 09:22:09 -0600
Subject: [C++-sig] Python Exported Method that Takes a Boost::Function Object
Message-ID: <708326ee1002010722u5f033562i3799c5fa0295aa87@mail.gmail.com>

I have a method that takes a boost function object as so

bool createTimer( boost::int64_t interval, boost::function< bool() >
function, bool recurring = false )

that I would like to call in python.  I tried exporting the class in Py++
but it did not look like it does anything special with that argument.
However, when I try to use this from python I get

Boost.Python.ArgumentError: Python argument types in
    createTimer( int, method, bool )
did not match C++ signature:
    createTimer( unsigned long interval, class boost::function<bool
__cdecl(void)> function, bool recurring=False)

that is with the export code

.def( "createTimer", &Class::createTimer, ( bp::arg( "interval" ), bp::arg(
"function" ), bp::arg( "recurring" ) = false ) )

Py++ recommends exporting it like

     { //::Class::createTimer

            typedef ::boost::int32_t ( ::Class::*createTimer_function_type
)( ::boost::uint32_t,::boost::function< bool ()() >,bool ) ;

            Class_exposer.def(
                "createTimer"
                , createTimer_function_type( &::Class::createTimer )
                , ( bp::arg("interval"), bp::arg("function"),
bp::arg("recurring")=(bool)(false) ) );

        }

which btw there is an error here in Py++, ::boost::function< bool ()() >
should be ::boost::function< bool () >

that code also fails in python with the same error.

I have seen the page here
http://wiki.python.org/moin/boost.python/HowTo#boost.functionobjects about
boost function objects but I do not think that is exactly what I am looking
for.

If there an easy way to get python to work with this function definition?
Or am I trying to do something stupid?

with this code I can do createTimer( 3, boost::bind( &longFunc, arg1, arg2 )
) which is extremely convenient.  I would like to have the same ability in
python.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100201/e4298912/attachment.htm>

From anders.e.e.wallin at gmail.com  Mon Feb  1 18:07:45 2010
From: anders.e.e.wallin at gmail.com (Anders Wallin)
Date: Mon, 1 Feb 2010 19:07:45 +0200
Subject: [C++-sig] OpenMP and boost-python?
Message-ID: <5ce676911002010907m6a753c42vd04d021fa541faeb@mail.gmail.com>

Hello group,
I have C++ code which uses OpenMP to parallellize some algorithms.
When wrapped with boost-python only one thread seems to run at a time,
and there is no speedup (as seen with pure C++) when using multiple
core machines.

The FAQ says this is pretty much expected, but hints at a possible patch:
http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/faq.html#threadsupport

How difficult is it to make boost-python play nice with OpenMP? Anyone
done it? Is this ever going to be a feature of boost-python, or always
going to require special patching and hacking?

thanks,
Anders

From seefeld at sympatico.ca  Mon Feb  1 18:23:43 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Mon, 01 Feb 2010 12:23:43 -0500
Subject: [C++-sig] OpenMP and boost-python?
In-Reply-To: <5ce676911002010907m6a753c42vd04d021fa541faeb@mail.gmail.com>
References: <5ce676911002010907m6a753c42vd04d021fa541faeb@mail.gmail.com>
Message-ID: <4B670E1F.7020807@sympatico.ca>

On 02/01/2010 12:07 PM, Anders Wallin wrote:
> Hello group,
> I have C++ code which uses OpenMP to parallellize some algorithms.
> When wrapped with boost-python only one thread seems to run at a time,
> and there is no speedup (as seen with pure C++) when using multiple
> core machines.
>
> The FAQ says this is pretty much expected, but hints at a possible patch:
> http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/faq.html#threadsupport
>
> How difficult is it to make boost-python play nice with OpenMP? Anyone
> done it? Is this ever going to be a feature of boost-python, or always
> going to require special patching and hacking?
>    

I'm not sure in how much boost.python would actually need to care about 
this. Presumably, some OpenMP support needs to be provided by the 
runtime library (such as the thread management), so I would expect the 
Python binary (or whatever your main application is) to be responsible 
for initializing the right runtime library / support.

What compiler are you using ? Does it provide some documentation on what 
needs to be done to activate OpenMP at runtime ?

Regards,
         Stefan


-- 

       ...ich hab' noch einen Koffer in Berlin...


From troy at resophonic.com  Mon Feb  1 18:31:01 2010
From: troy at resophonic.com (troy d. straszheim)
Date: Mon, 01 Feb 2010 12:31:01 -0500
Subject: [C++-sig] OpenMP and boost-python?
In-Reply-To: <5ce676911002010907m6a753c42vd04d021fa541faeb@mail.gmail.com>
References: <5ce676911002010907m6a753c42vd04d021fa541faeb@mail.gmail.com>
Message-ID: <4B670FD5.1050904@resophonic.com>

Anders Wallin wrote:
> Hello group,
> I have C++ code which uses OpenMP to parallellize some algorithms.
> When wrapped with boost-python only one thread seems to run at a time,
> and there is no speedup (as seen with pure C++) when using multiple
> core machines.
> 
> The FAQ says this is pretty much expected, but hints at a possible patch:
> http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/faq.html#threadsupport
> 
> How difficult is it to make boost-python play nice with OpenMP? Anyone
> done it? Is this ever going to be a feature of boost-python, or always
> going to require special patching and hacking?
> 

Boost.MPI has python bindings, you could look at those for ideas.

-t



From anders.e.e.wallin at gmail.com  Mon Feb  1 18:58:41 2010
From: anders.e.e.wallin at gmail.com (Anders Wallin)
Date: Mon, 1 Feb 2010 19:58:41 +0200
Subject: [C++-sig] OpenMP and boost-python?
In-Reply-To: <4B670E1F.7020807@sympatico.ca>
References: <5ce676911002010907m6a753c42vd04d021fa541faeb@mail.gmail.com>
	<4B670E1F.7020807@sympatico.ca>
Message-ID: <5ce676911002010958w72e81b62oe0347cf67d81e87@mail.gmail.com>

>> How difficult is it to make boost-python play nice with OpenMP? Anyone
>> done it? Is this ever going to be a feature of boost-python, or always
>> going to require special patching and hacking?
> I'm not sure in how much boost.python would actually need to care about
> this. Presumably, some OpenMP support needs to be provided by the runtime
> library (such as the thread management), so I would expect the Python binary
> (or whatever your main application is) to be responsible for initializing
> the right runtime library / support.
> What compiler are you using ? Does it provide some documentation on what
> needs to be done to activate OpenMP at runtime ?

I am using GCC 4.4.1
The pure C++ code with OpenMP compiles and runs fine using many
threads. There is almost a linear speedup of trivially parallel
for-loops.

When this same code is wrapped with boost-python and called from
Python that speedup disappears, i.e. the Python GIL(?, or something
similar), forces the C++ code to run in a single thread, or only one
thread to execute at a time.

MPI is clearly one option, but requires a re-write of the whole code,
and results in code that does not resemble the single-thread code
much. I would like to use OpenMP, since it's a "for dummies" way of
parallellizing things.

Any other suggestions for writing parallel C++ code which is called
from Python ??

AW

From seefeld at sympatico.ca  Mon Feb  1 19:15:45 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Mon, 01 Feb 2010 13:15:45 -0500
Subject: [C++-sig] OpenMP and boost-python?
In-Reply-To: <5ce676911002010958w72e81b62oe0347cf67d81e87@mail.gmail.com>
References: <5ce676911002010907m6a753c42vd04d021fa541faeb@mail.gmail.com>	<4B670E1F.7020807@sympatico.ca>
	<5ce676911002010958w72e81b62oe0347cf67d81e87@mail.gmail.com>
Message-ID: <4B671A51.8000404@sympatico.ca>

On 02/01/2010 12:58 PM, Anders Wallin wrote:
>>> How difficult is it to make boost-python play nice with OpenMP? Anyone
>>> done it? Is this ever going to be a feature of boost-python, or always
>>> going to require special patching and hacking?
>>>        
>> I'm not sure in how much boost.python would actually need to care about
>> this. Presumably, some OpenMP support needs to be provided by the runtime
>> library (such as the thread management), so I would expect the Python binary
>> (or whatever your main application is) to be responsible for initializing
>> the right runtime library / support.
>> What compiler are you using ? Does it provide some documentation on what
>> needs to be done to activate OpenMP at runtime ?
>>      
> I am using GCC 4.4.1
> The pure C++ code with OpenMP compiles and runs fine using many
> threads. There is almost a linear speedup of trivially parallel
> for-loops.
>
> When this same code is wrapped with boost-python and called from
> Python that speedup disappears, i.e. the Python GIL(?, or something
> similar), forces the C++ code to run in a single thread, or only one
> thread to execute at a time.
>    

I'm not convinced of this. It is true that there may only ever be one 
thread accessing the Python runtime (thus the GIL to enforce it). But 
you may certainly have as many threads doing other work as you want. And 
if your OpenMP-enabled code is pure C++, with no hooks into the Python 
runtime, then I don't see why that may not work.

Again: If your OpenMP-using code and the Python-exposed API are well 
isolated, I don't think either needs to know about the other.

     Stefan


-- 

       ...ich hab' noch einen Koffer in Berlin...


From anders.e.e.wallin at gmail.com  Mon Feb  1 19:26:51 2010
From: anders.e.e.wallin at gmail.com (Anders Wallin)
Date: Mon, 1 Feb 2010 20:26:51 +0200
Subject: [C++-sig] OpenMP and boost-python?
In-Reply-To: <4B671A51.8000404@sympatico.ca>
References: <5ce676911002010907m6a753c42vd04d021fa541faeb@mail.gmail.com>
	<4B670E1F.7020807@sympatico.ca>
	<5ce676911002010958w72e81b62oe0347cf67d81e87@mail.gmail.com>
	<4B671A51.8000404@sympatico.ca>
Message-ID: <5ce676911002011026g7ab1cda7kc172a415c5fc8b6b@mail.gmail.com>

> I'm not convinced of this. It is true that there may only ever be one thread
> accessing the Python runtime (thus the GIL to enforce it). But you may
> certainly have as many threads doing other work as you want. And if your
> OpenMP-enabled code is pure C++, with no hooks into the Python runtime, then
> I don't see why that may not work.
> Again: If your OpenMP-using code and the Python-exposed API are well
> isolated, I don't think either needs to know about the other.

Thanks, in fact I now got my trivial example working with a good
speedup! For the real code I need to look at isolating C++/Python.

AW

From roman.yakovenko at gmail.com  Mon Feb  1 20:40:56 2010
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Mon, 1 Feb 2010 21:40:56 +0200
Subject: [C++-sig] Python Exported Method that Takes a Boost::Function
	Object
In-Reply-To: <708326ee1002010722u5f033562i3799c5fa0295aa87@mail.gmail.com>
References: <708326ee1002010722u5f033562i3799c5fa0295aa87@mail.gmail.com>
Message-ID: <7465b6171002011140q11fd5ca8x86291eb21503c92@mail.gmail.com>

On Mon, Feb 1, 2010 at 5:22 PM, Charles Solar <charlessolar at gmail.com> wrote:
> I have a method that takes a boost function object as so
>
> bool createTimer( boost::int64_t interval, boost::function< bool() >
> function, bool recurring = false )
>
> that I would like to call in python.? I tried exporting the class in Py++
> but it did not look like it does anything special with that argument.

Py++ generates the wrong code in this situation. It definitely could
do better job.

>
> which btw there is an error here in Py++, ::boost::function< bool ()() >
> should be ::boost::function< bool () >

Hmm. This is how gccxml reports the class name. I will take a look on this bug.

> that code also fails in python with the same error.
>
> I have seen the page here
> http://wiki.python.org/moin/boost.python/HowTo#boost.functionobjects about
> boost function objects but I do not think that is exactly what I am looking
> for.
>
> If there an easy way to get python to work with this function definition?
> Or am I trying to do something stupid?

No, I would call this a "challenge".

The following is just an untested idea:

1. create the following class:

class callback_wrapper_t{
    callback_wrapper_t( boost::python::object callable ){ store it in the class}

   bool operator()( bool ){
        call the function
   }
}

2. create a wrapper

bool createTimerWrapper( boost::int64_t interval,
boost::python::object function, bool recurring = false ){
    createTimer( interval, boost::function<bool>( callback_wrapper_t(
function ) ), recurring )
}

3. Register the createTimerWrapper

I think this could work. May be there is a better way. If you have a
lot of functions like this and the idea works, I can help you to
generate it with Py++.


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

From charlessolar at gmail.com  Mon Feb  1 21:18:01 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Mon, 1 Feb 2010 14:18:01 -0600
Subject: [C++-sig] Python Exported Method that Takes a Boost::Function
	Object
In-Reply-To: <7465b6171002011140q11fd5ca8x86291eb21503c92@mail.gmail.com>
References: <708326ee1002010722u5f033562i3799c5fa0295aa87@mail.gmail.com> 
	<7465b6171002011140q11fd5ca8x86291eb21503c92@mail.gmail.com>
Message-ID: <708326ee1002011218x7f655504h39a380271480e7c4@mail.gmail.com>

Nice! A little bit of cleaning up and editing and that worked like a charm,
thanks

Here is what I had to do:

struct timer_func_wrapper_t
{
    timer_func_wrapper_t( bp::object callable ) : _callable( callable ) {}

    bool operator()()
    {
        PyGILState_STATE gstate = PyGILState_Ensure();
        bool ret = _callable();
        PyGILState_Release( gstate );
        return ret;
    }

    bp::object _callable;
};

boost::int32_t createTimerWrapper( Class* class, boost::uint64_t interval,
bp::object function, bool recurring = false )
{
    return class->createTimer( interval, boost::function<bool ()>(
timer_func_wrapper_t( function ) ), recurring );
}

and then in the class def:

.def( "createTimer", &createTimerWrapper, ( bp::arg( "interval" ), bp::arg(
"function" ), bp::arg( "recurring" ) = false ) )

I think I kind of cheated with the wrapper function, the this pointer just
fills in the Class* hole.. I was actually surprised that it worked so
easily.

Anyway now I can do magic like

import MyLib
import time

def callMePls():
    print( "Hello world" )
    return True

class = MyLib.Class()

class.createTimer( 3, callMePls )

time.sleep( 1 )

Works great!

About the Py++ stuff though, I mainly use Py++ to see how I should export
something I am unsure of.  Which is why I mentioned that Py++ did not do
anything special, since I was unsure how to export this in the first place.
I type up and maintain my real export code though, so no rush on Py++
support, although I bet its as easy as generating a patch or something..
Anyway, great tool, great advice, thanks again.

On Mon, Feb 1, 2010 at 1:40 PM, Roman Yakovenko
<roman.yakovenko at gmail.com>wrote:

> On Mon, Feb 1, 2010 at 5:22 PM, Charles Solar <charlessolar at gmail.com>
> wrote:
> > I have a method that takes a boost function object as so
> >
> > bool createTimer( boost::int64_t interval, boost::function< bool() >
> > function, bool recurring = false )
> >
> > that I would like to call in python.  I tried exporting the class in Py++
> > but it did not look like it does anything special with that argument.
>
> Py++ generates the wrong code in this situation. It definitely could
> do better job.
>
> >
> > which btw there is an error here in Py++, ::boost::function< bool ()() >
> > should be ::boost::function< bool () >
>
> Hmm. This is how gccxml reports the class name. I will take a look on this
> bug.
>
> > that code also fails in python with the same error.
> >
> > I have seen the page here
> > http://wiki.python.org/moin/boost.python/HowTo#boost.functionobjectsabout
> > boost function objects but I do not think that is exactly what I am
> looking
> > for.
> >
> > If there an easy way to get python to work with this function definition?
> > Or am I trying to do something stupid?
>
> No, I would call this a "challenge".
>
> The following is just an untested idea:
>
> 1. create the following class:
>
> class callback_wrapper_t{
>    callback_wrapper_t( boost::python::object callable ){ store it in the
> class}
>
>   bool operator()( bool ){
>        call the function
>   }
> }
>
> 2. create a wrapper
>
> bool createTimerWrapper( boost::int64_t interval,
> boost::python::object function, bool recurring = false ){
>    createTimer( interval, boost::function<bool>( callback_wrapper_t(
> function ) ), recurring )
> }
>
> 3. Register the createTimerWrapper
>
> I think this could work. May be there is a better way. If you have a
> lot of functions like this and the idea works, I can help you to
> generate it with Py++.
>
>
> --
> Roman Yakovenko
> C++ Python language binding
> http://www.language-binding.net/
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100201/b99e870a/attachment.htm>

From pascal.briet at optimprocess.com  Tue Feb  2 12:46:17 2010
From: pascal.briet at optimprocess.com (Pascal Briet)
Date: Tue, 2 Feb 2010 12:46:17 +0100
Subject: [C++-sig] PyGILState_Release with multithread
Message-ID: <202B7D9ED9E14E25B3835169A94816AF@pcdellDev>

Hello,

 

After a few days of headache, I think the best way is to share this
problematic


 

I have :

- a main Python script

- a C module imported and called from Python  (thanks to boost ::python)

- A dozen of threads created by a C function  (boost ::thread)

- These threads, in a C main loop, call regularly a Python function.

 

 

Here is the critical part of the C code, with the Python GIL management :

 

boost::mutex      _pyMutex;

PyGILState_STATE  _gstate;

 

_pyMutex.lock();

_gstate = PyGILState_Ensure();

_pyMutex.unlock();

 

// Interprets Python bytecodes with boost::python

 

_pyMutex.lock();

PyGILState_Release(_gstate);

_pyMutex.unlock();

 

 

This part is multi-threaded, and it seems that the PyGILState_* functions do
not like it.

I have the following error with PyGILState_Release : ? This thread state
must be current when releasing ?

 

I think that when 2 threads enter PyGILState_Ensure, the second one is
considered as the current one. When the first one end
  it is not current. A
quick look to the code of the Python API seems to confirm it.

 

 

So, the only solution I found is to lock the whole call to Python :

 

boost::mutex      _pyMutex; 

PyGILState_STATE  _gstate;

 

_pyMutex.lock();

_gstate = PyGILState_Ensure();

 

// Interprets Python bytecodes with boost::python

 

PyGILState_Release(_gstate);

_pyMutex.unlock();

 

 

Everything is ok with this solution, except that
  only one thread at a time
can call a Python method.

When 2 threads ask for a Python call, if the first one is a blocking
operation, the second one will wait
  it is so bad !

 

Do you have any solution ?

 

Nb : I use Python 2.5.2

 

Thanks for your help.

 

 

_____

 

Briet Pascal

Ing?nieur d'?tudes

 

+33 (0)1 47 42 10 55

 

OptimProcess

32 rue Tronchet - 75009 Paris 
 <http://www.optimprocess.com/> www.optimprocess.com

 





 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100202/800ebcea/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 2121 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100202/800ebcea/attachment.jpeg>

From peoro.noob at gmail.com  Tue Feb  2 16:53:16 2010
From: peoro.noob at gmail.com (peoro)
Date: Tue, 2 Feb 2010 16:53:16 +0100
Subject: [C++-sig] [Py++] C++ containers and Python sequences
Message-ID: <1c7ed23c1002020753q3fe6e3acp25e631c081da108b@mail.gmail.com>

Hello,
let me start by thanking everybody who's behind Boost.Python and Py++,
these projects are really awesome.

I read that Py++ (and/or Boost.Python) has a native support for some
C++ containers, and that others can be added using indexing suites [
http://www.language-binding.net/pyplusplus/documentation/containers.html
]; was thus expecting a native conversions between C++ containers and
Python sequences.
For example, by exposing bindings for the following C++ function:
void f( const std::vector<int> &v );
I supposed the next Python expression would have been valid:
mypackage.f( [1,2,3] );
but actually it's not.
Only std::string gets automagically converted to/from str, other
supported containers can be explicitly casted to Python sequences (eg:
passing an exposed std::vector<int> to Python's "list", it works), but
not the other way around.

I've been trying to play with pyplusplus.module_builder, but cannot
find a way to allow native conversions, and google isn't being of help
either.
Is it possible to achieve native conversions, and, if it is, why is it
not done by default?

Thanks!
Paolo Giangrandi

From Matthew.Scouten at tradingtechnologies.com  Tue Feb  2 18:30:37 2010
From: Matthew.Scouten at tradingtechnologies.com (Matthew Scouten (TT))
Date: Tue, 2 Feb 2010 11:30:37 -0600
Subject: [C++-sig] PyGILState_Release with multithread
In-Reply-To: <202B7D9ED9E14E25B3835169A94816AF@pcdellDev>
References: <202B7D9ED9E14E25B3835169A94816AF@pcdellDev>
Message-ID: <32490DFF7774554A85D65D23A9F0F9380C4DEE5F@chiex01>

First of all, the PyGILState_* functions are acquiring and releasing the "Global Interpreter _Lock_" Your _pyMutex is redundant. 

Here are the rules that govern the GIL:

1)      Only one thread at a time can hold the GIL

2)      Any thread that is touching python data, python code, or any part of the 'terp must hold the GIL

 

The python part of your program WILL be single threaded. There is nothing you can do about that. 

 

From: cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org [mailto:cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org] On Behalf Of Pascal Briet
Sent: Tuesday, February 02, 2010 5:46 AM
To: cplusplus-sig at python.org
Subject: [C++-sig] PyGILState_Release with multithread

 

Hello,

 

After a few days of headache, I think the best way is to share this problematic...

 

I have :

- a main Python script

- a C module imported and called from Python  (thanks to boost ::python)

- A dozen of threads created by a C function  (boost ::thread)

- These threads, in a C main loop, call regularly a Python function.

 

 

Here is the critical part of the C code, with the Python GIL management :

 

boost::mutex      _pyMutex;

PyGILState_STATE  _gstate;

 

_pyMutex.lock();

_gstate = PyGILState_Ensure();

_pyMutex.unlock();

 

// Interprets Python bytecodes with boost::python

 

_pyMutex.lock();

PyGILState_Release(_gstate);

_pyMutex.unlock();

 

 

This part is multi-threaded, and it seems that the PyGILState_* functions do not like it.

I have the following error with PyGILState_Release : ? This thread state must be current when releasing ?

 

I think that when 2 threads enter PyGILState_Ensure, the second one is considered as the current one. When the first one end...  it is not current. A quick look to the code of the Python API seems to confirm it.

 

 

So, the only solution I found is to lock the whole call to Python :

 

boost::mutex      _pyMutex; 

PyGILState_STATE  _gstate;

 

_pyMutex.lock();

_gstate = PyGILState_Ensure();

 

// Interprets Python bytecodes with boost::python

 

PyGILState_Release(_gstate);

_pyMutex.unlock();

 

 

Everything is ok with this solution, except that...  only one thread at a time can call a Python method.

When 2 threads ask for a Python call, if the first one is a blocking operation, the second one will wait...  it is so bad !

 

Do you have any solution ?

 

Nb : I use Python 2.5.2

 

Thanks for your help.

 

 

_____

 

Briet Pascal

Ing?nieur d'?tudes

 

+33 (0)1 47 42 10 55

 

OptimProcess

32 rue Tronchet - 75009 Paris 
www.optimprocess.com <http://www.optimprocess.com/> 

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100202/caf3ed57/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 2121 bytes
Desc: image001.jpg
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100202/caf3ed57/attachment-0001.jpeg>

From renatox at gmail.com  Tue Feb  2 19:53:26 2010
From: renatox at gmail.com (Renato Araujo)
Date: Tue, 2 Feb 2010 15:53:26 -0300
Subject: [C++-sig] PyGILState_Release with multithread
In-Reply-To: <32490DFF7774554A85D65D23A9F0F9380C4DEE5F@chiex01>
References: <202B7D9ED9E14E25B3835169A94816AF@pcdellDev>
	<32490DFF7774554A85D65D23A9F0F9380C4DEE5F@chiex01>
Message-ID: <95291a81002021053tf7dc7c8m9c697f47d5a36aca@mail.gmail.com>

Remember of use PyEval_InitThreads on your module initialization.

Renato Araujo Oliveira Filho



On Tue, Feb 2, 2010 at 2:30 PM, Matthew Scouten (TT) <
Matthew.Scouten at tradingtechnologies.com> wrote:

>  First of all, the PyGILState_* functions are acquiring and releasing the
> ?Global Interpreter _*Lock*_? Your _pyMutex is redundant.
>
> Here are the rules that govern the GIL:
>
> 1)      Only one thread at a time can hold the GIL
>
> 2)      Any thread that is touching python data, python code, or any part
> of the ?terp must hold the GIL
>
>
>
> The python part of your program WILL be single threaded. There is nothing
> you can do about that.
>
>
>
> *From:* cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com@
> python.org [mailto:cplusplus-sig-bounces+matthew.scouten<cplusplus-sig-bounces%2Bmatthew.scouten>
> =tradingtechnologies.com at python.org] *On Behalf Of *Pascal Briet
> *Sent:* Tuesday, February 02, 2010 5:46 AM
> *To:* cplusplus-sig at python.org
> *Subject:* [C++-sig] PyGILState_Release with multithread
>
>
>
> Hello,
>
>
>
> After a few days of headache, I think the best way is to share this
> problematic?
>
>
>
> I have :
>
> - a main Python script
>
> - a C module imported and called from Python  (thanks to boost ::python)
>
> - A dozen of threads created by a C function  (boost ::thread)
>
> - These threads, in a C main loop, call regularly a Python function.
>
>
>
>
>
> Here is the critical part of the C code, with the Python GIL management :
>
>
>
> boost::mutex      _pyMutex;
>
> PyGILState_STATE  _gstate;
>
>
>
> _pyMutex.lock();
>
> _gstate = PyGILState_Ensure();
>
> _pyMutex.unlock();
>
>
>
> // Interprets Python bytecodes with boost::python
>
>
>
> _pyMutex.lock();
>
> PyGILState_Release(_gstate);
>
> _pyMutex.unlock();
>
>
>
>
>
> This part is multi-threaded, and it seems that the PyGILState_* functions
> do not like it.
>
> I have the following error with PyGILState_Release : ? This thread state
> must be current when releasing ?
>
>
>
> I think that when 2 threads enter PyGILState_Ensure, the second one is
> considered as the current one. When the first one end?  it is not current. A
> quick look to the code of the Python API seems to confirm it.
>
>
>
>
>
> So, the only solution I found is to lock the whole call to Python :
>
>
>
> boost::mutex      _pyMutex;
>
> PyGILState_STATE  _gstate;
>
>
>
> _pyMutex.lock();
>
> _gstate = PyGILState_Ensure();
>
>
>
> // Interprets Python bytecodes with boost::python
>
>
>
> PyGILState_Release(_gstate);
>
> _pyMutex.unlock();
>
>
>
>
>
> Everything is ok with this solution, except that?  only one thread at a
> time can call a Python method.
>
> When 2 threads ask for a Python call, if the first one is a blocking
> operation, the second one will wait?  it is so bad !
>
>
>
> Do you have any solution ?
>
>
>
> Nb : I use Python 2.5.2
>
>
>
> Thanks for your help.
>
>
>
>
>
> *_____*
>
>
>
> *Briet Pascal*
>
> Ing?nieur d'?tudes
>
>
>
> +33 (0)1 47 42 10 55
>
>
>
> OptimProcess
>
> 32 rue Tronchet - 75009 Paris
> www.optimprocess.com
>
>
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100202/0fe084e5/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 2121 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100202/0fe084e5/attachment.jpeg>

From roman.yakovenko at gmail.com  Tue Feb  2 20:20:44 2010
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Tue, 2 Feb 2010 21:20:44 +0200
Subject: [C++-sig] [Py++] C++ containers and Python sequences
In-Reply-To: <1c7ed23c1002020753q3fe6e3acp25e631c081da108b@mail.gmail.com>
References: <1c7ed23c1002020753q3fe6e3acp25e631c081da108b@mail.gmail.com>
Message-ID: <7465b6171002021120s72bd9b4aj583ec669aeef1c3b@mail.gmail.com>

On Tue, Feb 2, 2010 at 5:53 PM, peoro <peoro.noob at gmail.com> wrote:
> Hello,
> let me start by thanking everybody who's behind Boost.Python and Py++,
> these projects are really awesome.
>
> I read that Py++ (and/or Boost.Python) has a native support for some
> C++ containers, and that others can be added using indexing suites [
> http://www.language-binding.net/pyplusplus/documentation/containers.html
> ]; was thus expecting a native conversions between C++ containers and
> Python sequences.
> For example, by exposing bindings for the following C++ function:
> void f( const std::vector<int> &v );
> I supposed the next Python expression would have been valid:
> mypackage.f( [1,2,3] );
> but actually it's not.
> Only std::string gets automagically converted to/from str, other
> supported containers can be explicitly casted to Python sequences (eg:
> passing an exposed std::vector<int> to Python's "list", it works), but
> not the other way around.

This subject was recently discussed on pygccxml mailing list:

http://sourceforge.net/mailarchive/forum.php?thread_name=BAY127-W4F7525C7D12CD96F134EF8F640%40phx.gbl&forum_name=pygccxml-development


> I've been trying to play with pyplusplus.module_builder, but cannot
> find a way to allow native conversions, and google isn't being of help
> either.
> Is it possible to achieve native conversions, and, if it is, why is it
> not done by default?

Hope the posted link  will clarify few things for you.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

From pascal.briet at optimprocess.com  Wed Feb  3 10:55:03 2010
From: pascal.briet at optimprocess.com (Pascal Briet)
Date: Wed, 3 Feb 2010 10:55:03 +0100
Subject: [C++-sig] Cplusplus-sig Digest, Vol 17, Issue 4
In-Reply-To: <mailman.8943.1265132456.28904.cplusplus-sig@python.org>
References: <mailman.8943.1265132456.28904.cplusplus-sig@python.org>
Message-ID: <367EF2F1FA094B2ABB1CDE388FE46A6A@pcdellDev>

Thanks to Renato, the solution was just... PyEval_InitThreads ! 
Thanks for your help !


Matthew, you're right, there is just one thread running at a time in Python.

But, if my understanding of the Python API is correct, more than one thread
can be registered to the Python Interpreter (PyThreadState). Calling
PyGILState_Ensure function creates a PyThreadState.
Next, the Python API will switch its current thread every 100 bytecodes
(this value is the default of sys.setcheckinterval function). There are
never two system threads running at the same time in the Python API, but at
least... they are alternated.


One proof is that PyGILState_Ensure is _not_ blocking the second thread
during the execution of the first thread in Python. It probably waits a few
microseconds, as I explain below, but the 2 threads enter the "critical"
Python part together. I checked that :

static int count = 0;

 
_gstate = PyGILState_Ensure();
if (count != 0)
	// In my code, I use a thread-safe stream.
	std::cout << "2 threads together" << std::endl; 
count++;

 // Interprets Python bytecodes with boost::python

count--;
PyGILState_Release(_gstate);


The message "2 threads together" appears. Without PyEval_InitThreads, it
crashes just after.


You are right about the locks, they are not needed.

Let's detail what happens :

a) Thread 1 : GIL_Ensure(New PyThread_State registered, GIL lock)
b) Thread 1 : Executing Python through the Interpreter   |   Thread 2 :
GIL_Ensure (waiting for GIL)
c) Thread 1 : Executing the 100th bytecode : Python Interpreter Releases the
GIL. Switch to Thread 2.
d) Thread 1 : waiting for GIL  | Thread 2 : Executing Python
e) Thread 2 : 100th bytecode : Switch to Thread 1.
f) Thread 1 : Executing Python | Thread 2 : Waiting for GIL
etc.


-- 
BRIET Pascal


------------------------------

Remember of use PyEval_InitThreads on your module initialization.

Renato Araujo Oliveira Filho


------------------------------

Message: 3
Date: Tue, 2 Feb 2010 11:30:37 -0600
From: "Matthew Scouten (TT)" <Matthew.Scouten at tradingtechnologies.com>
To: "Development of Python/C++ integration" <cplusplus-sig at python.org>
Subject: Re: [C++-sig] PyGILState_Release with multithread
Message-ID: <32490DFF7774554A85D65D23A9F0F9380C4DEE5F at chiex01>
Content-Type: text/plain; charset="iso-8859-1"

First of all, the PyGILState_* functions are acquiring and releasing the
"Global Interpreter _Lock_" Your _pyMutex is redundant. 

Here are the rules that govern the GIL:

1)      Only one thread at a time can hold the GIL

2)      Any thread that is touching python data, python code, or any part of
the 'terp must hold the GIL

 

The python part of your program WILL be single threaded. There is nothing
you can do about that. 

 

From:
cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org
[mailto:cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python
.org] On Behalf Of Pascal Briet
Sent: Tuesday, February 02, 2010 5:46 AM
To: cplusplus-sig at python.org
Subject: [C++-sig] PyGILState_Release with multithread

 

Hello,

 

After a few days of headache, I think the best way is to share this
problematic...

 

I have :

- a main Python script

- a C module imported and called from Python  (thanks to boost ::python)

- A dozen of threads created by a C function  (boost ::thread)

- These threads, in a C main loop, call regularly a Python function.

 

 

Here is the critical part of the C code, with the Python GIL management :

 

boost::mutex      _pyMutex;

PyGILState_STATE  _gstate;

 

_pyMutex.lock();

_gstate = PyGILState_Ensure();

_pyMutex.unlock();

 

// Interprets Python bytecodes with boost::python

 

_pyMutex.lock();

PyGILState_Release(_gstate);

_pyMutex.unlock();

 

 

This part is multi-threaded, and it seems that the PyGILState_* functions do
not like it.

I have the following error with PyGILState_Release : ? This thread state
must be current when releasing ?

 

I think that when 2 threads enter PyGILState_Ensure, the second one is
considered as the current one. When the first one end...  it is not current.
A quick look to the code of the Python API seems to confirm it.

 

 

So, the only solution I found is to lock the whole call to Python :

 

boost::mutex      _pyMutex; 

PyGILState_STATE  _gstate;

 

_pyMutex.lock();

_gstate = PyGILState_Ensure();

 

// Interprets Python bytecodes with boost::python

 

PyGILState_Release(_gstate);

_pyMutex.unlock();

 

 

Everything is ok with this solution, except that...  only one thread at a
time can call a Python method.

When 2 threads ask for a Python call, if the first one is a blocking
operation, the second one will wait...  it is so bad !

 

Do you have any solution ?

 

Nb : I use Python 2.5.2

 

Thanks for your help.

 




From nitroamos at gmail.com  Wed Feb  3 21:24:13 2010
From: nitroamos at gmail.com (Amos Anderson)
Date: Wed, 3 Feb 2010 12:24:13 -0800
Subject: [C++-sig] how do i interrupt a C++ extension?
Message-ID: <9e910971002031224i4ce2fb37i3543ec74500fe6e5@mail.gmail.com>

Hello --

I've got a python script with C++ extensions. Some of my extensions
take a long time to complete, and I don't want to wait for them to
finish when I'm debugging stuff. However, when I do Ctrl-C in my
terminal, it's completely ignored. So it looks like python is trapping
the signal, but apparently can't do anything with it until the
extension returns control to the python script. I guess ideally,
Ctrl-C would kill the extension and return control to python,
generating an exception, but I'd also be ok if Ctrl-C killed the
python script too.

I've been googling around, but can't figure out how this seemingly
simple (and desired) task is accomplished. Anybody know how to do it?
Right now, the only solution is Ctrl-Z and kill %1 so I guess that
works for now...

thanks!

Amos.

From Matthew.Scouten at tradingtechnologies.com  Wed Feb  3 22:22:11 2010
From: Matthew.Scouten at tradingtechnologies.com (Matthew Scouten (TT))
Date: Wed, 3 Feb 2010 15:22:11 -0600
Subject: [C++-sig] how do i interrupt a C++ extension?
In-Reply-To: <9e910971002031224i4ce2fb37i3543ec74500fe6e5@mail.gmail.com>
References: <9e910971002031224i4ce2fb37i3543ec74500fe6e5@mail.gmail.com>
Message-ID: <32490DFF7774554A85D65D23A9F0F9380C5851BC@chiex01>

There is no solution. This is a problem inherent in the way python
handles the GIL, c extensions, and Signals. Details here:
http://www.dabeaz.com/python/GIL.pdf see slide 22 

-----Original Message-----
From:
cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org
[mailto:cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at py
thon.org] On Behalf Of Amos Anderson
Sent: Wednesday, February 03, 2010 2:24 PM
To: Development of Python/C++ integration
Subject: [C++-sig] how do i interrupt a C++ extension?

Hello --

I've got a python script with C++ extensions. Some of my extensions
take a long time to complete, and I don't want to wait for them to
finish when I'm debugging stuff. However, when I do Ctrl-C in my
terminal, it's completely ignored. So it looks like python is trapping
the signal, but apparently can't do anything with it until the
extension returns control to the python script. I guess ideally,
Ctrl-C would kill the extension and return control to python,
generating an exception, but I'd also be ok if Ctrl-C killed the
python script too.

I've been googling around, but can't figure out how this seemingly
simple (and desired) task is accomplished. Anybody know how to do it?
Right now, the only solution is Ctrl-Z and kill %1 so I guess that
works for now...

thanks!

Amos.
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig at python.org
http://mail.python.org/mailman/listinfo/cplusplus-sig

From rwgk at yahoo.com  Wed Feb  3 22:42:21 2010
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Wed, 3 Feb 2010 13:42:21 -0800 (PST)
Subject: [C++-sig] how do i interrupt a C++ extension?
In-Reply-To: <9e910971002031224i4ce2fb37i3543ec74500fe6e5@mail.gmail.com>
References: <9e910971002031224i4ce2fb37i3543ec74500fe6e5@mail.gmail.com>
Message-ID: <124463.50752.qm@web111412.mail.gq1.yahoo.com>

It would be a nice feature to have, but in 8+ years working with Boost.Python I never truly needed it.

I figure if your extension runs a long time it must have some layers of loops. If you can modify the
extension code, I'd reorganize it to move the outer loop into Python. If the extension is a function,
I'd turn it into a class like this:

  calc = calculation()
  while (calc.is_not_finished()):
    calc.inner_loop()

If you figure out how to make Ctrl-C work portably (!), post it here!

Ralf


----- Original Message ----
From: Amos Anderson <nitroamos at gmail.com>
To: Development of Python/C++ integration <cplusplus-sig at python.org>
Sent: Wed, February 3, 2010 12:24:13 PM
Subject: [C++-sig] how do i interrupt a C++ extension?

Hello --

I've got a python script with C++ extensions. Some of my extensions
take a long time to complete, and I don't want to wait for them to
finish when I'm debugging stuff. However, when I do Ctrl-C in my
terminal, it's completely ignored. So it looks like python is trapping
the signal, but apparently can't do anything with it until the
extension returns control to the python script. I guess ideally,
Ctrl-C would kill the extension and return control to python,
generating an exception, but I'd also be ok if Ctrl-C killed the
python script too.

I've been googling around, but can't figure out how this seemingly
simple (and desired) task is accomplished. Anybody know how to do it?
Right now, the only solution is Ctrl-Z and kill %1 so I guess that
works for now...

thanks!

Amos.
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig at python.org
http://mail.python.org/mailman/listinfo/cplusplus-sig


From gjcarneiro at gmail.com  Thu Feb  4 00:57:29 2010
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Wed, 3 Feb 2010 23:57:29 +0000
Subject: [C++-sig] how do i interrupt a C++ extension?
In-Reply-To: <124463.50752.qm@web111412.mail.gq1.yahoo.com>
References: <9e910971002031224i4ce2fb37i3543ec74500fe6e5@mail.gmail.com>
	<124463.50752.qm@web111412.mail.gq1.yahoo.com>
Message-ID: <a467ca4f1002031557q178dd19bnd15f98abf2f440c5@mail.gmail.com>

On Wed, Feb 3, 2010 at 9:42 PM, Ralf W. Grosse-Kunstleve <rwgk at yahoo.com>wrote:

> It would be a nice feature to have, but in 8+ years working with
> Boost.Python I never truly needed it.
>
> I figure if your extension runs a long time it must have some layers of
> loops. If you can modify the
> extension code, I'd reorganize it to move the outer loop into Python. If
> the extension is a function,
> I'd turn it into a class like this:
>
>  calc = calculation()
>  while (calc.is_not_finished()):
>    calc.inner_loop()
>
> If you figure out how to make Ctrl-C work portably (!), post it here!
>

I wouldn't advise this solution exactly like this.  It may work, but is
going to be slow.  Moving loops into Python makes the computation probably a
lot slower.

The solution I adopted for the NS-3 simulator (ns3.Simulator.Run()) was kind
of hybrid.  There is an outer loop, and there is an API to run a single
iteration of that loop.  The ns3.Simulator.Run() wrapper[1] runs iterations
of the loop the following way: we run iterations while there are events to
process; every 100 iterations we 1) acquire the GIL, 2) call
PyErr_CheckSignals() and check the result of PyErr_Occurred(), 3) release
the GIL again.

[1] See _wrap_Simulator_Run in
http://code.nsnam.org/ns-3-dev/file/0ca25e25b116/bindings/python/ns3module_helpers.cc


> Ralf
>
>
> ----- Original Message ----
> From: Amos Anderson <nitroamos at gmail.com>
> To: Development of Python/C++ integration <cplusplus-sig at python.org>
> Sent: Wed, February 3, 2010 12:24:13 PM
> Subject: [C++-sig] how do i interrupt a C++ extension?
>
> Hello --
>
> I've got a python script with C++ extensions. Some of my extensions
> take a long time to complete, and I don't want to wait for them to
> finish when I'm debugging stuff. However, when I do Ctrl-C in my
> terminal, it's completely ignored. So it looks like python is trapping
> the signal, but apparently can't do anything with it until the
> extension returns control to the python script. I guess ideally,
> Ctrl-C would kill the extension and return control to python,
> generating an exception, but I'd also be ok if Ctrl-C killed the
> python script too.
>
> I've been googling around, but can't figure out how this seemingly
> simple (and desired) task is accomplished. Anybody know how to do it?
> Right now, the only solution is Ctrl-Z and kill %1 so I guess that
> works for now...
>
> thanks!
>
> Amos.
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>



-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100203/6215cc67/attachment.htm>

From anders.e.e.wallin at gmail.com  Thu Feb  4 19:22:58 2010
From: anders.e.e.wallin at gmail.com (Anders Wallin)
Date: Thu, 4 Feb 2010 20:22:58 +0200
Subject: [C++-sig] accessing nontrivial types
Message-ID: <5ce676911002041022h344f3024v58e8e1d8994dfca6@mail.gmail.com>

Hello group,
simple types (int, float, etc.) seem to transfer without problem to
C++ and back to Python, but whenever I try returning something
nontrivial from C++ I get something like this:
>>> a = myclass.Myclass()
>>> print a.i
42
>>> print a.f
3.14159
>>> print a.ilist
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: No Python class registered for C++ class std::list<int,
std::allocator<int> >


The code for this example is here:
http://pastebin.com/m4c45a88

I tried looking in the documentation but was not able to understand
what to do about this error. Any examples out there?

thanks,
AW

From Matthew.Scouten at tradingtechnologies.com  Thu Feb  4 19:45:20 2010
From: Matthew.Scouten at tradingtechnologies.com (Matthew Scouten (TT))
Date: Thu, 4 Feb 2010 12:45:20 -0600
Subject: [C++-sig] accessing nontrivial types
In-Reply-To: <5ce676911002041022h344f3024v58e8e1d8994dfca6@mail.gmail.com>
References: <5ce676911002041022h344f3024v58e8e1d8994dfca6@mail.gmail.com>
Message-ID: <32490DFF7774554A85D65D23A9F0F9380C5854B5@chiex01>

The stl containers are not handled by default. You will have to manually
expose it. You have a few options here:
	1) the vector_indexing_suite might work for a std::list. Give it
a try.
	2) Manually expose the function the you care about. 
	3) have a couple of get/set functions as a property that copies
your std::list to and from a bp::list. 

-----Original Message-----
From:
cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org
[mailto:cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at py
thon.org] On Behalf Of Anders Wallin
Sent: Thursday, February 04, 2010 12:23 PM
To: Development of Python/C++ integration
Subject: [C++-sig] accessing nontrivial types

Hello group,
simple types (int, float, etc.) seem to transfer without problem to
C++ and back to Python, but whenever I try returning something
nontrivial from C++ I get something like this:
>>> a = myclass.Myclass()
>>> print a.i
42
>>> print a.f
3.14159
>>> print a.ilist
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: No Python class registered for C++ class std::list<int,
std::allocator<int> >


The code for this example is here:
http://pastebin.com/m4c45a88

I tried looking in the documentation but was not able to understand
what to do about this error. Any examples out there?

thanks,
AW
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig at python.org
http://mail.python.org/mailman/listinfo/cplusplus-sig

From anders.e.e.wallin at gmail.com  Thu Feb  4 21:12:01 2010
From: anders.e.e.wallin at gmail.com (Anders Wallin)
Date: Thu, 4 Feb 2010 22:12:01 +0200
Subject: [C++-sig] accessing nontrivial types
In-Reply-To: <32490DFF7774554A85D65D23A9F0F9380C5854B5@chiex01>
References: <5ce676911002041022h344f3024v58e8e1d8994dfca6@mail.gmail.com>
	<32490DFF7774554A85D65D23A9F0F9380C5854B5@chiex01>
Message-ID: <5ce676911002041212o64475d45r93547e18069bc7be@mail.gmail.com>

are there examples of how to do this somewhere?
for something like list<int>, MyClass   and list<MyClass>

thanks,
AW

> ? ? ? ?1) the vector_indexing_suite might work for a std::list. Give it
> a try.
> ? ? ? ?2) Manually expose the function the you care about.
> ? ? ? ?3) have a couple of get/set functions as a property that copies
> your std::list to and from a bp::list.

From murrayc at murrayc.com  Thu Feb  4 21:16:41 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Thu, 04 Feb 2010 21:16:41 +0100
Subject: [C++-sig] Getting a b::p::object for a PyObject and vice-versa.
Message-ID: <1265314601.8686.15.camel@murrayc-x61>

Can someone confirm that this is correct:

1.
To get a boost::python::object that wraps an existing PyObject* (when
you get a PyObject from a C function not wrapped in boost::python), can
someone confirm that this is correct:
Either:
a)
  PyObject* c_object = get_the_c_object(); //returns a reference.
  boost::python::handle<> handle(c_object);
  boost::python::object cpp_object(handle);
or
b)
  PyObject* c_object = get_the_c_object(); //returns no extra reference.
  boost::python::handle<> handle(boost::python::borrowed(cObject));
  boost::python::object cpp_object(handle);

The code for b) can be simplified by doing 
  boost::python::object cpp_object(boost::python::borrowed(cObject))
but I don't see a way to simplify a)

I also see allow_null here, but like borrowed, it's not documented:
http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/handle.html#allow_null-spec
so I can only guess at its purpose.

In general I wish this was much simpler and stated clearly somewhere.


2.
How can I get the underlying PyObject* from a boost::python::object
(when I need it to call a C function not wrapped in boost::python)?

I guessed at this, but it doesn't seem to be working for me:
  boost::python::extract<PyObject*> extractor(cpp_object);
  if(extractor.check())
  {
    PyObject* c_object = extractor;
  }
  


-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From Matthew.Scouten at tradingtechnologies.com  Thu Feb  4 21:25:44 2010
From: Matthew.Scouten at tradingtechnologies.com (Matthew Scouten (TT))
Date: Thu, 4 Feb 2010 14:25:44 -0600
Subject: [C++-sig] accessing nontrivial types
In-Reply-To: <5ce676911002041212o64475d45r93547e18069bc7be@mail.gmail.com>
References: <5ce676911002041022h344f3024v58e8e1d8994dfca6@mail.gmail.com><32490DFF7774554A85D65D23A9F0F9380C5854B5@chiex01>
	<5ce676911002041212o64475d45r93547e18069bc7be@mail.gmail.com>
Message-ID: <32490DFF7774554A85D65D23A9F0F9380C585565@chiex01>

1)
class_< std::list<Thingy> >("ListOfThingys")
            .def(vector_indexing_suite< std::list<Thingy> >() )
2)
This is really too complex to show quickly. Just read the c++ header and do the same thing to list that you did to myclass. 

3)
bp::list getter (const MyClass& self) 
{
	bp::list L
	//move the values in self.ilist to L
	return L
} 
void setter (const MyClass& self, bp::list L) 
{
	//move the values in L to self.ilist
}

class_<Myclass>("Myclass")
        .def_readwrite("i", &Myclass::i)
        .def_readwrite("f", &Myclass::f)
        .add_property ("ilist", &getter, &setter)
;

That's the gist of it. I am not sure if 1 works and 2 will be difficult, so I recommend 3


-----Original Message-----
From: cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org [mailto:cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org] On Behalf Of Anders Wallin
Sent: Thursday, February 04, 2010 2:12 PM
To: Development of Python/C++ integration
Subject: Re: [C++-sig] accessing nontrivial types

are there examples of how to do this somewhere?
for something like list<int>, MyClass   and list<MyClass>

thanks,
AW

> ? ? ? ?1) the vector_indexing_suite might work for a std::list. Give it
> a try.
> ? ? ? ?2) Manually expose the function the you care about.
> ? ? ? ?3) have a couple of get/set functions as a property that copies
> your std::list to and from a bp::list.
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig at python.org
http://mail.python.org/mailman/listinfo/cplusplus-sig

From anders.e.e.wallin at gmail.com  Fri Feb  5 16:01:40 2010
From: anders.e.e.wallin at gmail.com (Anders Wallin)
Date: Fri, 5 Feb 2010 17:01:40 +0200
Subject: [C++-sig] accessing nontrivial types
In-Reply-To: <32490DFF7774554A85D65D23A9F0F9380C585565@chiex01>
References: <5ce676911002041022h344f3024v58e8e1d8994dfca6@mail.gmail.com>
	<32490DFF7774554A85D65D23A9F0F9380C5854B5@chiex01>
	<5ce676911002041212o64475d45r93547e18069bc7be@mail.gmail.com>
	<32490DFF7774554A85D65D23A9F0F9380C585565@chiex01>
Message-ID: <5ce676911002050701w159320betbee95fd0f5a0a260@mail.gmail.com>

> 3)
> bp::list getter (const MyClass& self)
> {
> ? ? ? ?bp::list L
> ? ? ? ?//move the values in self.ilist to L
> ? ? ? ?return L
> }
> void setter (const MyClass& self, bp::list L)
> {
> ? ? ? ?//move the values in L to self.ilist
> }
>
> class_<Myclass>("Myclass")
> ? ? ? ?.def_readwrite("i", &Myclass::i)
> ? ? ? ?.def_readwrite("f", &Myclass::f)
> ? ? ? ?.add_property ("ilist", &getter, &setter)
> ;
>
> That's the gist of it. I am not sure if 1 works and 2 will be difficult, so I recommend 3

Thanks, this works. Incidentally, "getter" seems to be a defined
function somewhere else, so I had better luck naming it "listgetter"
or something similar.

AW

From rwgk at yahoo.com  Fri Feb  5 18:47:17 2010
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Fri, 5 Feb 2010 09:47:17 -0800 (PST)
Subject: [C++-sig] Getting a b::p::object for a PyObject and vice-versa.
In-Reply-To: <1265314601.8686.15.camel@murrayc-x61>
References: <1265314601.8686.15.camel@murrayc-x61>
Message-ID: <173917.58406.qm@web111415.mail.gq1.yahoo.com>

What you do looks right.

bp::object(bp::handle<>(c_object))

may be a little shorter in some places. (Watch out for compilers confusing your code with
function declarations.)

bpobj.ptr() will give you the PyObject* directly.
bphdl.get() does the same for handle<>.

(Improvements are always welcome, but a complete set of patches is a lot of work
and I doubt even wrapping the entire Python API will make much of a practical
difference.)

Ralf



----- Original Message ----
From: Murray Cumming <murrayc at murrayc.com>
To: cplusplus-sig at python.org
Sent: Thu, February 4, 2010 12:16:41 PM
Subject: [C++-sig] Getting a b::p::object for a PyObject and vice-versa.

Can someone confirm that this is correct:

1.
To get a boost::python::object that wraps an existing PyObject* (when
you get a PyObject from a C function not wrapped in boost::python), can
someone confirm that this is correct:
Either:
a)
  PyObject* c_object = get_the_c_object(); //returns a reference.
  boost::python::handle<> handle(c_object);
  boost::python::object cpp_object(handle);
or
b)
  PyObject* c_object = get_the_c_object(); //returns no extra reference.
  boost::python::handle<> handle(boost::python::borrowed(cObject));
  boost::python::object cpp_object(handle);

The code for b) can be simplified by doing 
  boost::python::object cpp_object(boost::python::borrowed(cObject))
but I don't see a way to simplify a)

I also see allow_null here, but like borrowed, it's not documented:
http://www.boost.org/doc/libs/1_41_0/libs/python/doc/v2/handle.html#allow_null-spec
so I can only guess at its purpose.

In general I wish this was much simpler and stated clearly somewhere.


2.
How can I get the underlying PyObject* from a boost::python::object
(when I need it to call a C function not wrapped in boost::python)?

I guessed at this, but it doesn't seem to be working for me:
  boost::python::extract<PyObject*> extractor(cpp_object);
  if(extractor.check())
  {
    PyObject* c_object = extractor;
  }

From murrayc at murrayc.com  Sat Feb  6 14:39:54 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Sat, 06 Feb 2010 14:39:54 +0100
Subject: [C++-sig] Getting the wrapped C++ instance from a
	boost::python::object
Message-ID: <1265463594.2545.2.camel@murrayc-x61>

If I have a boost::python::object that I know contains a PyObject that
wraps an instance of my C++ class, MyClass, can I get a pointer to that
MyClass instance, so I can call its C++ methods?

-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From seefeld at sympatico.ca  Sat Feb  6 18:54:11 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Sat, 06 Feb 2010 12:54:11 -0500
Subject: [C++-sig] Getting the wrapped C++ instance from
	a	boost::python::object
In-Reply-To: <1265463594.2545.2.camel@murrayc-x61>
References: <1265463594.2545.2.camel@murrayc-x61>
Message-ID: <4B6DACC3.3090803@sympatico.ca>

On 02/06/2010 08:39 AM, Murray Cumming wrote:
> If I have a boost::python::object that I know contains a PyObject that
> wraps an instance of my C++ class, MyClass, can I get a pointer to that
> MyClass instance, so I can call its C++ methods?
>    

http://www.boost.org/doc/libs/1_41_0/libs/python/doc/tutorial/doc/html/python/object.html#python.extracting_c___objects


     Stefan

-- 

       ...ich hab' noch einen Koffer in Berlin...


From murrayc at murrayc.com  Sat Feb  6 20:57:42 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Sat, 06 Feb 2010 20:57:42 +0100
Subject: [C++-sig] Getting the wrapped C++ instance from a
 boost::python::object
In-Reply-To: <4B6DACC3.3090803@sympatico.ca>
References: <1265463594.2545.2.camel@murrayc-x61>
	<4B6DACC3.3090803@sympatico.ca>
Message-ID: <1265486262.3613.2.camel@murrayc-x61>

On Sat, 2010-02-06 at 12:54 -0500, Stefan Seefeld wrote:
> On 02/06/2010 08:39 AM, Murray Cumming wrote:
> > If I have a boost::python::object that I know contains a PyObject that
> > wraps an instance of my C++ class, MyClass, can I get a pointer to that
> > MyClass instance, so I can call its C++ methods?
> >    
> 
> http://www.boost.org/doc/libs/1_41_0/libs/python/doc/tutorial/doc/html/python/object.html#python.extracting_c___objects

So you are suggesting that I can do 
  boost::python::extract<MyClass*> extractor(obj);
  if(extractor.check())
  {
    MyClass* myclass = extractor;
  }
?

Thanks.

-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From nitroamos at gmail.com  Sat Feb  6 23:06:50 2010
From: nitroamos at gmail.com (Amos Anderson)
Date: Sat, 6 Feb 2010 14:06:50 -0800
Subject: [C++-sig] how do i interrupt a C++ extension?
Message-ID: <9e910971002061406n150fc1d6ucb4a07bf8ac61cb8@mail.gmail.com>

Thanks for the responses! It sounds like there's just no way to send a
signal to C++.

Moving loops from C++ to Python around is not really a solution for us
because we need to be moving them in the other direction if they're to
be moved at all. This is molecular simulation code, so some of the
extensions will probably run for hours/days... For example, a dynamics
simulation needs to know if you're planning on killing it so that it
can print out the latest iteration. Other times, I'm just debugging
it, and I don't need it to run to completion.

Maybe the best solution would be to have the C++ code check some file
or something. Then I could write in the file "die gracefully" and it
would respond when it reads it.



Amos.



On Thu, Feb 4, 2010 at 3:00 AM,  <cplusplus-sig-request at python.org> wrote:
> Send Cplusplus-sig mailing list submissions to
> ? ? ? ?cplusplus-sig at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> ? ? ? ?http://mail.python.org/mailman/listinfo/cplusplus-sig
> or, via email, send a message with subject or body 'help' to
> ? ? ? ?cplusplus-sig-request at python.org
>
> You can reach the person managing the list at
> ? ? ? ?cplusplus-sig-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Cplusplus-sig digest..."
>
>
> Today's Topics:
>
> ? 1. how do i interrupt a C++ extension? (Amos Anderson)
> ? 2. Re: how do i interrupt a C++ extension? (Matthew Scouten (TT))
> ? 3. Re: how do i interrupt a C++ extension? (Ralf W. Grosse-Kunstleve)
> ? 4. Re: how do i interrupt a C++ extension? (Gustavo Carneiro)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 3 Feb 2010 12:24:13 -0800
> From: Amos Anderson <nitroamos at gmail.com>
> To: "Development of Python/C++ integration" <cplusplus-sig at python.org>
> Subject: [C++-sig] how do i interrupt a C++ extension?
> Message-ID:
> ? ? ? ?<9e910971002031224i4ce2fb37i3543ec74500fe6e5 at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hello --
>
> I've got a python script with C++ extensions. Some of my extensions
> take a long time to complete, and I don't want to wait for them to
> finish when I'm debugging stuff. However, when I do Ctrl-C in my
> terminal, it's completely ignored. So it looks like python is trapping
> the signal, but apparently can't do anything with it until the
> extension returns control to the python script. I guess ideally,
> Ctrl-C would kill the extension and return control to python,
> generating an exception, but I'd also be ok if Ctrl-C killed the
> python script too.
>
> I've been googling around, but can't figure out how this seemingly
> simple (and desired) task is accomplished. Anybody know how to do it?
> Right now, the only solution is Ctrl-Z and kill %1 so I guess that
> works for now...
>
> thanks!
>
> Amos.
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 3 Feb 2010 15:22:11 -0600
> From: "Matthew Scouten (TT)" <Matthew.Scouten at tradingtechnologies.com>
> To: "Development of Python/C++ integration" <cplusplus-sig at python.org>
> Subject: Re: [C++-sig] how do i interrupt a C++ extension?
> Message-ID: <32490DFF7774554A85D65D23A9F0F9380C5851BC at chiex01>
> Content-Type: text/plain; ? ? ? charset="US-ASCII"
>
> There is no solution. This is a problem inherent in the way python
> handles the GIL, c extensions, and Signals. Details here:
> http://www.dabeaz.com/python/GIL.pdf see slide 22
>
> -----Original Message-----
> From:
> cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at python.org
> [mailto:cplusplus-sig-bounces+matthew.scouten=tradingtechnologies.com at py
> thon.org] On Behalf Of Amos Anderson
> Sent: Wednesday, February 03, 2010 2:24 PM
> To: Development of Python/C++ integration
> Subject: [C++-sig] how do i interrupt a C++ extension?
>
> Hello --
>
> I've got a python script with C++ extensions. Some of my extensions
> take a long time to complete, and I don't want to wait for them to
> finish when I'm debugging stuff. However, when I do Ctrl-C in my
> terminal, it's completely ignored. So it looks like python is trapping
> the signal, but apparently can't do anything with it until the
> extension returns control to the python script. I guess ideally,
> Ctrl-C would kill the extension and return control to python,
> generating an exception, but I'd also be ok if Ctrl-C killed the
> python script too.
>
> I've been googling around, but can't figure out how this seemingly
> simple (and desired) task is accomplished. Anybody know how to do it?
> Right now, the only solution is Ctrl-Z and kill %1 so I guess that
> works for now...
>
> thanks!
>
> Amos.
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 3 Feb 2010 13:42:21 -0800 (PST)
> From: "Ralf W. Grosse-Kunstleve" <rwgk at yahoo.com>
> To: Development of Python/C++ integration <cplusplus-sig at python.org>
> Subject: Re: [C++-sig] how do i interrupt a C++ extension?
> Message-ID: <124463.50752.qm at web111412.mail.gq1.yahoo.com>
> Content-Type: text/plain; charset=us-ascii
>
> It would be a nice feature to have, but in 8+ years working with Boost.Python I never truly needed it.
>
> I figure if your extension runs a long time it must have some layers of loops. If you can modify the
> extension code, I'd reorganize it to move the outer loop into Python. If the extension is a function,
> I'd turn it into a class like this:
>
> ?calc = calculation()
> ?while (calc.is_not_finished()):
> ? ?calc.inner_loop()
>
> If you figure out how to make Ctrl-C work portably (!), post it here!
>
> Ralf
>
>
> ----- Original Message ----
> From: Amos Anderson <nitroamos at gmail.com>
> To: Development of Python/C++ integration <cplusplus-sig at python.org>
> Sent: Wed, February 3, 2010 12:24:13 PM
> Subject: [C++-sig] how do i interrupt a C++ extension?
>
> Hello --
>
> I've got a python script with C++ extensions. Some of my extensions
> take a long time to complete, and I don't want to wait for them to
> finish when I'm debugging stuff. However, when I do Ctrl-C in my
> terminal, it's completely ignored. So it looks like python is trapping
> the signal, but apparently can't do anything with it until the
> extension returns control to the python script. I guess ideally,
> Ctrl-C would kill the extension and return control to python,
> generating an exception, but I'd also be ok if Ctrl-C killed the
> python script too.
>
> I've been googling around, but can't figure out how this seemingly
> simple (and desired) task is accomplished. Anybody know how to do it?
> Right now, the only solution is Ctrl-Z and kill %1 so I guess that
> works for now...
>
> thanks!
>
> Amos.
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
>
>
> ------------------------------
>
> Message: 4
> Date: Wed, 3 Feb 2010 23:57:29 +0000
> From: Gustavo Carneiro <gjcarneiro at gmail.com>
> To: "Development of Python/C++ integration" <cplusplus-sig at python.org>
> Subject: Re: [C++-sig] how do i interrupt a C++ extension?
> Message-ID:
> ? ? ? ?<a467ca4f1002031557q178dd19bnd15f98abf2f440c5 at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Wed, Feb 3, 2010 at 9:42 PM, Ralf W. Grosse-Kunstleve <rwgk at yahoo.com>wrote:
>
>> It would be a nice feature to have, but in 8+ years working with
>> Boost.Python I never truly needed it.
>>
>> I figure if your extension runs a long time it must have some layers of
>> loops. If you can modify the
>> extension code, I'd reorganize it to move the outer loop into Python. If
>> the extension is a function,
>> I'd turn it into a class like this:
>>
>> ?calc = calculation()
>> ?while (calc.is_not_finished()):
>> ? ?calc.inner_loop()
>>
>> If you figure out how to make Ctrl-C work portably (!), post it here!
>>
>
> I wouldn't advise this solution exactly like this. ?It may work, but is
> going to be slow. ?Moving loops into Python makes the computation probably a
> lot slower.
>
> The solution I adopted for the NS-3 simulator (ns3.Simulator.Run()) was kind
> of hybrid. ?There is an outer loop, and there is an API to run a single
> iteration of that loop. ?The ns3.Simulator.Run() wrapper[1] runs iterations
> of the loop the following way: we run iterations while there are events to
> process; every 100 iterations we 1) acquire the GIL, 2) call
> PyErr_CheckSignals() and check the result of PyErr_Occurred(), 3) release
> the GIL again.
>
> [1] See _wrap_Simulator_Run in
> http://code.nsnam.org/ns-3-dev/file/0ca25e25b116/bindings/python/ns3module_helpers.cc
>
>
>> Ralf
>>
>>
>> ----- Original Message ----
>> From: Amos Anderson <nitroamos at gmail.com>
>> To: Development of Python/C++ integration <cplusplus-sig at python.org>
>> Sent: Wed, February 3, 2010 12:24:13 PM
>> Subject: [C++-sig] how do i interrupt a C++ extension?
>>
>> Hello --
>>
>> I've got a python script with C++ extensions. Some of my extensions
>> take a long time to complete, and I don't want to wait for them to
>> finish when I'm debugging stuff. However, when I do Ctrl-C in my
>> terminal, it's completely ignored. So it looks like python is trapping
>> the signal, but apparently can't do anything with it until the
>> extension returns control to the python script. I guess ideally,
>> Ctrl-C would kill the extension and return control to python,
>> generating an exception, but I'd also be ok if Ctrl-C killed the
>> python script too.
>>
>> I've been googling around, but can't figure out how this seemingly
>> simple (and desired) task is accomplished. Anybody know how to do it?
>> Right now, the only solution is Ctrl-Z and kill %1 so I guess that
>> works for now...
>>
>> thanks!
>>
>> Amos.
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig at python.org
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig at python.org
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>
>
>
>
> --
> Gustavo J. A. M. Carneiro
> INESC Porto, Telecommunications and Multimedia Unit
> "The universe is always one step beyond logic." -- Frank Herbert
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100203/6215cc67/attachment-0001.htm>
>
> ------------------------------
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
> End of Cplusplus-sig Digest, Vol 17, Issue 7
> ********************************************
>



-- 
~<>~<>~<>~<>~<>~<>~<>~<>~
Amos G. Anderson
+1-626-399-8958 (cell)

From troy at resophonic.com  Sat Feb  6 23:44:30 2010
From: troy at resophonic.com (troy d. straszheim)
Date: Sat, 06 Feb 2010 17:44:30 -0500
Subject: [C++-sig] how do i interrupt a C++ extension?
In-Reply-To: <9e910971002061406n150fc1d6ucb4a07bf8ac61cb8@mail.gmail.com>
References: <9e910971002061406n150fc1d6ucb4a07bf8ac61cb8@mail.gmail.com>
Message-ID: <4B6DF0CE.4050701@resophonic.com>

Amos Anderson wrote:
> Thanks for the responses! It sounds like there's just no way to send a
> signal to C++.
> 

Please watch the top-posting...

> Moving loops from C++ to Python around is not really a solution for us
> because we need to be moving them in the other direction if they're to
> be moved at all. This is molecular simulation code, so some of the
> extensions will probably run for hours/days... For example, a dynamics
> simulation needs to know if you're planning on killing it so that it
> can print out the latest iteration. Other times, I'm just debugging
> it, and I don't need it to run to completion.
> 
> Maybe the best solution would be to have the C++ code check some file
> or something. Then I could write in the file "die gracefully" and it
> would respond when it reads it.

I think you should reread Gustavo's suggestion.

-t


From j.reid at mail.cryst.bbk.ac.uk  Sun Feb  7 01:22:05 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Sun, 07 Feb 2010 00:22:05 +0000
Subject: [C++-sig] how do i interrupt a C++ extension?
In-Reply-To: <9e910971002061406n150fc1d6ucb4a07bf8ac61cb8@mail.gmail.com>
References: <9e910971002061406n150fc1d6ucb4a07bf8ac61cb8@mail.gmail.com>
Message-ID: <hkl13c$81e$1@ger.gmane.org>



Amos Anderson wrote:
> Thanks for the responses! It sounds like there's just no way to send a
> signal to C++.
> 
> Moving loops from C++ to Python around is not really a solution for us
> because we need to be moving them in the other direction if they're to
> be moved at all. This is molecular simulation code, so some of the
> extensions will probably run for hours/days... For example, a dynamics
> simulation needs to know if you're planning on killing it so that it
> can print out the latest iteration. Other times, I'm just debugging
> it, and I don't need it to run to completion.
> 
> Maybe the best solution would be to have the C++ code check some file
> or something. Then I could write in the file "die gracefully" and it
> would respond when it reads it.
> 

Another straightforward way is to pass a python callback function into 
your C++ code. This can also be used for progress reporting. A ctrl-C 
will be caught whenever the callback is called even if the callback does 
nothing.

John.


From murrayc at murrayc.com  Sun Feb  7 11:22:12 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Sun, 07 Feb 2010 11:22:12 +0100
Subject: [C++-sig] Executing python code from C++
Message-ID: <1265538132.2404.8.camel@murrayc-x61>

In Glom, to execute arbitrary Python code, I use PyRun_String() to parse
the Python code, PyDict_GetItemString() and PyObject_Call() to get a
callable object for that code, and then PyObject_CallObject() to
actually execute the code and get a return value. This was the result of
experimentation and code archeology:
http://git.gnome.org/browse/glom/tree/glom/python_embed/glom_python.cc?h=glom-1-12#n269

Is there anything in boost::python that wraps this, so I can make the
code nicer?

-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From murrayc at murrayc.com  Sun Feb  7 11:47:08 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Sun, 07 Feb 2010 11:47:08 +0100
Subject: [C++-sig] Checking for or null PyObject* or PyNone
Message-ID: <1265539628.2404.16.camel@murrayc-x61>

Doing this
  boost::python::object obj = get_some_object();
  if(obj)
  {
     //do something
  }
seems to check a bool value inside the underlying PyObject, though I
guess that could throw an exception if it doesn't contain actually
contain a bool. Is that right?
The reference documentation is not very clear about this:
http://www.boost.org/doc/libs/1_42_0/libs/python/doc/v2/object.html


I had wrongly guessed that it would check if the underlying PyObject*
was 0. I did this because I had wrapped a PyObject* in a
boost::python::object for convenience, without checking for 0 first.

Is there any correct way to check, other than doing
  if(obj.ptr())
?


And what's the correct way to check for PyNone other than 
  if (obj == boost::python::object())
?

-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From seefeld at sympatico.ca  Sun Feb  7 14:03:43 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Sun, 07 Feb 2010 08:03:43 -0500
Subject: [C++-sig] Executing python code from C++
In-Reply-To: <1265538132.2404.8.camel@murrayc-x61>
References: <1265538132.2404.8.camel@murrayc-x61>
Message-ID: <4B6EBA2F.6040807@sympatico.ca>

On 02/07/2010 05:22 AM, Murray Cumming wrote:
> In Glom, to execute arbitrary Python code, I use PyRun_String() to parse
> the Python code, PyDict_GetItemString() and PyObject_Call() to get a
> callable object for that code, and then PyObject_CallObject() to
> actually execute the code and get a return value. This was the result of
> experimentation and code archeology:
> http://git.gnome.org/browse/glom/tree/glom/python_embed/glom_python.cc?h=glom-1-12#n269
>
> Is there anything in boost::python that wraps this, so I can make the
> code nicer?
>    

Yes. Please read the Boost.Python documentation, specifically on Embedding:
http://www.boost.org/doc/libs/1_42_0/libs/python/doc/tutorial/doc/html/python/embedding.html

     Stefan

-- 

       ...ich hab' noch einen Koffer in Berlin...


From murrayc at murrayc.com  Sun Feb  7 14:29:53 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Sun, 07 Feb 2010 14:29:53 +0100
Subject: [C++-sig] Executing python code from C++
In-Reply-To: <4B6EBA2F.6040807@sympatico.ca>
References: <1265538132.2404.8.camel@murrayc-x61>
	<4B6EBA2F.6040807@sympatico.ca>
Message-ID: <1265549393.2404.35.camel@murrayc-x61>

On Sun, 2010-02-07 at 08:03 -0500, Stefan Seefeld wrote:
> On 02/07/2010 05:22 AM, Murray Cumming wrote:
> > In Glom, to execute arbitrary Python code, I use PyRun_String() to parse
> > the Python code, PyDict_GetItemString() and PyObject_Call() to get a
> > callable object for that code, and then PyObject_CallObject() to
> > actually execute the code and get a return value. This was the result of
> > experimentation and code archeology:
> > http://git.gnome.org/browse/glom/tree/glom/python_embed/glom_python.cc?h=glom-1-12#n269
> >
> > Is there anything in boost::python that wraps this, so I can make the
> > code nicer?
> >    
> 
> Yes. Please read the Boost.Python documentation, specifically on Embedding:
> http://www.boost.org/doc/libs/1_42_0/libs/python/doc/tutorial/doc/html/python/embedding.html

Thanks.

So, I guess I can use boost::python::exec() to call code that defines a
Python function.

But how can I get a boost::python::object for the (callable object of)
the function? In C, I'm using PyDict_GetItemString() for that.

-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From seefeld at sympatico.ca  Sun Feb  7 15:51:42 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Sun, 07 Feb 2010 09:51:42 -0500
Subject: [C++-sig] Executing python code from C++
In-Reply-To: <1265549393.2404.35.camel@murrayc-x61>
References: <1265538132.2404.8.camel@murrayc-x61>	<4B6EBA2F.6040807@sympatico.ca>
	<1265549393.2404.35.camel@murrayc-x61>
Message-ID: <4B6ED37E.6080401@sympatico.ca>

On 02/07/2010 08:29 AM, Murray Cumming wrote:
> So, I guess I can use boost::python::exec() to call code that defines a
> Python function.
>    

bpl::exec() executes a chunk of Python code, no matter what it contains.

> But how can I get a boost::python::object for the (callable object of)
> the function? In C, I'm using PyDict_GetItemString() for that.
>    

Sorry, I don't understand the question. Can you give an example of what 
you want to do ? May be you want to "exec" some python code that defines 
a function, which you then want to extract and call later ?
That may look like this:

   // Retrieve the main module
   bpl::object main = bpl::import("__main__");
   // Retrieve the main module's namespace
   bpl::object global(main.attr("__dict__"));
   // Define the 'embedded' Python code...
   std::string py_code = "def greeting(): print 'hello world !'";
   // ...and execute it.
   bpl::object result = bpl::exec(py_code, global, global);
   // Extract the function
   bpl::object greeting = global["greeting"];
   // Call it
   greeting();

You may combine this with the other techniques outlines previously. For 
example, your Python code may define types based on previously exported 
C++ types (Python classes derived from C++ classes, say), and then 
extract them to gain back C++ references to base classes. There are 
endless possibilities to play with such hybrid code... :-)

     Stefan

-- 

       ...ich hab' noch einen Koffer in Berlin...


From gjcarneiro at gmail.com  Sun Feb  7 19:40:06 2010
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Sun, 7 Feb 2010 18:40:06 +0000
Subject: [C++-sig] ANN: PyBindGen 0.14
Message-ID: <a467ca4f1002071040l783a7b85g84bd42ed43de2b9b@mail.gmail.com>

PyBindGen is a Python module that is geared to generating C/C++ code that
binds a C/C++ library for Python. It does so without extensive use of either
C++ templates or C pre-processor macros. It has modular handling of C/C++
types, and can be easily extended with Python plugins. The generated code is
almost as clean as what a human programmer would write.

It can be downloaded from:

        http://code.google.com/p/pybindgen/

Bug reports should be filed here:

        https://bugs.launchpad.net/ <https://bugs.launchpad.net/pybindgen>
pybindgen <https://bugs.launchpad.net/pybindgen>

<https://bugs.launchpad.net/pybindgen>Documentation:

        http://packages.python.org/PyBindGen/

NEWS:

=== pybindgen 0.14 ===

    - Multiple inheritance support

    - Virtual methods can now be overridden with Method instead of _Method

    - Add annotation support for instance attributes

    - Benchmarks (vs Boost.Python, SWIG, and SIP)

    - New types supported: int16_t& and std::string* parameter types

    - Non-virtual protected methods are now also wrapped

    - Wrap enum pointer params

Note: this release marks the beginning of a transition to a minor change
in the API of the generated bindings.

Prior to PyBindGen version 0.14, the code generated to handle C++
virtual methods required Python user code to define a _foo method in
order to implement the virtual method foo.  Since 0.14, PyBindGen
changed so that virtual method foo is implemented in Python by
defining a method foo, i.e. no underscore prefix is needed anymore.
Setting pybindgen.settings.deprecated_virtuals to True will force the
old virtual method behaviour.  But this is really deprecated; newer
code should set pybindgen.settings.deprecated_virtuals to False.

In PyBindGen 0.14, if the option
pybindgen.settings.deprecated_virtuals is not set, PyBindGen emits a
warning and assumes the value of True in order to preserve backward
compatibility.  In PyBindGen 0.15, the default value of this option
will change to False, and in 0.16 the support for deprecated virtuals
will be removed.

This change was made to make the user code interface more intuitive,
and to align it with other Python bindings tools such as SIP,
Boost.Python, and SWIG.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, UTM, WiN, http://win.inescporto.pt/gjc
"The universe is always one step beyond logic." -- Frank Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100207/a81592eb/attachment.htm>

From ndbecker2 at gmail.com  Sun Feb  7 21:57:47 2010
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 07 Feb 2010 15:57:47 -0500
Subject: [C++-sig] ANN: PyBindGen 0.14
References: <a467ca4f1002071040l783a7b85g84bd42ed43de2b9b@mail.gmail.com>
Message-ID: <hkn9g9$ufe$1@ger.gmane.org>

PyBindGen shows an example wrapping STL containers.  What if I need to wrap 
my own containers?  Is there some generic container machinery?


From murrayc at murrayc.com  Sun Feb  7 22:51:45 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Sun, 07 Feb 2010 22:51:45 +0100
Subject: [C++-sig] Executing python code from C++
In-Reply-To: <4B6ED37E.6080401@sympatico.ca>
References: <1265538132.2404.8.camel@murrayc-x61>
	<4B6EBA2F.6040807@sympatico.ca> <1265549393.2404.35.camel@murrayc-x61>
	<4B6ED37E.6080401@sympatico.ca>
Message-ID: <1265579505.8678.2.camel@murrayc-x61>

On Sun, 2010-02-07 at 09:51 -0500, Stefan Seefeld wrote:
> 
> Sorry, I don't understand the question. Can you give an example of
> what 
> you want to do ? May be you want to "exec" some python code that
> defines 
> a function, which you then want to extract and call later ?
> That may look like this:
> 
>    // Retrieve the main module
>    bpl::object main = bpl::import("__main__");
>    // Retrieve the main module's namespace
>    bpl::object global(main.attr("__dict__"));
>    // Define the 'embedded' Python code...
>    std::string py_code = "def greeting(): print 'hello world !'";
>    // ...and execute it.
>    bpl::object result = bpl::exec(py_code, global, global);
>    // Extract the function
>    bpl::object greeting = global["greeting"]; 

Yes, this last line is what I figured out from some other example code.
It does work for me. Thanks.

I guess it make sense in terms of the weird Python C API, but it's all
far from obvious without knowledge of that.

Thanks again.

-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From murrayc at murrayc.com  Sun Feb  7 22:59:20 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Sun, 07 Feb 2010 22:59:20 +0100
Subject: [C++-sig] boost::python and Date/Time values
Message-ID: <1265579960.8678.5.camel@murrayc-x61>

Is there any easy way to get python Date, Time and DateTime values from
boost::python::objects? In C, I'm using PyDateTime_GET_YEAR(),
PyDateTime_GET_MONTH(), PyDateTime_GET_DAY(), etc, but it would be nice
to use some C++ API for this.

-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From seefeld at sympatico.ca  Sun Feb  7 23:15:30 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Sun, 07 Feb 2010 17:15:30 -0500
Subject: [C++-sig] Executing python code from C++
In-Reply-To: <1265579505.8678.2.camel@murrayc-x61>
References: <1265538132.2404.8.camel@murrayc-x61>	<4B6EBA2F.6040807@sympatico.ca>
	<1265549393.2404.35.camel@murrayc-x61>	<4B6ED37E.6080401@sympatico.ca>
	<1265579505.8678.2.camel@murrayc-x61>
Message-ID: <4B6F3B82.6090504@sympatico.ca>

On 02/07/2010 04:51 PM, Murray Cumming wrote:
> On Sun, 2010-02-07 at 09:51 -0500, Stefan Seefeld wrote:
>    
>> Sorry, I don't understand the question. Can you give an example of
>> what
>> you want to do ? May be you want to "exec" some python code that
>> defines
>> a function, which you then want to extract and call later ?
>> That may look like this:
>>
>>     // Retrieve the main module
>>     bpl::object main = bpl::import("__main__");
>>     // Retrieve the main module's namespace
>>     bpl::object global(main.attr("__dict__"));
>>     // Define the 'embedded' Python code...
>>     std::string py_code = "def greeting(): print 'hello world !'";
>>     // ...and execute it.
>>     bpl::object result = bpl::exec(py_code, global, global);
>>     // Extract the function
>>     bpl::object greeting = global["greeting"];
>>      
> Yes, this last line is what I figured out from some other example code.
> It does work for me. Thanks.
>
> I guess it make sense in terms of the weird Python C API, but it's all
> far from obvious without knowledge of that.
>    

I think this has nothing to do with the C API, but rather the way the 
"exec" builtin function is defined (i.e., within Python itself): It 
executes some Python code within the context of a 'global' and a 'local' 
namespace which may be provided to the "exec" call. It also returns an 
object, but that is in general rather useless (only useful for 
error-checkin inside the C API). The real effects of the invocation are 
to be found in the 'global' and 'local' namespaces, whose content may 
have changed...

> Thanks again.
>    

You are welcome.

         Stefan


-- 

       ...ich hab' noch einen Koffer in Berlin...


From gjcarneiro at gmail.com  Mon Feb  8 00:22:50 2010
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Sun, 7 Feb 2010 23:22:50 +0000
Subject: [C++-sig] ANN: PyBindGen 0.14
In-Reply-To: <hkn9g9$ufe$1@ger.gmane.org>
References: <a467ca4f1002071040l783a7b85g84bd42ed43de2b9b@mail.gmail.com>
	<hkn9g9$ufe$1@ger.gmane.org>
Message-ID: <a467ca4f1002071522m69c5c438gf4721956b5c09127@mail.gmail.com>

On Sun, Feb 7, 2010 at 8:57 PM, Neal Becker <ndbecker2 at gmail.com> wrote:

> PyBindGen shows an example wrapping STL containers.  What if I need to wrap
> my own containers?  Is there some generic container machinery?
>

Yes, there is some generic container machinery, although it currently is
duplicating the STL container machinery.  I hope some day to converge the
two, so that STL container support code simply uses the generic code, but I
have not found motivation to do it yet.

The generic container support can be used to add container powers to any
user class, and can be seen in the unit tests module, file tests/foo.h,
classes VectorLike, VectorLike2, and MapLike.

The class VectorLike registers methods for getitem/setitem/len slots (see
tests/foomodulegen.py):

    VectorLike.add_method('get_item', 'double', [Parameter.new('size_t',
'index')], custom_name='__getitem__')
    VectorLike.add_method('set_item', 'void', [Parameter.new('size_t',
'index'), Parameter.new('double', 'value')],
                          custom_name='__setitem__')
    VectorLike.add_method('get_len', 'size_t', [], custom_name='__len__')

The class VectorLike2 instead of the above slots defines begin/end methods
and uses iterators (see tests/foomodulegen_common.py).  The tp_iter slot is
defined:

    VectorLike2.add_container_traits(ReturnValue.new('double'),
begin_method='Begin', end_method='End', iterator_type='Iterator')

The class MapLike implements the "mapping protocol":

    MapLike.add_container_traits((ReturnValue.new('int'),
ReturnValue.new('double')),
                                 begin_method='Begin', end_method='End',
iterator_type='Iterator',
                                 is_mapping=True)


I'm sorry for the lack of examples and documentation.  Anyway, I hope this
helps.

Regards.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, UTM, WiN, http://win.inescporto.pt/gjc
"The universe is always one step beyond logic." -- Frank Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100207/49c524be/attachment.htm>

From talljimbo at gmail.com  Mon Feb  8 03:32:32 2010
From: talljimbo at gmail.com (Jim Bosch)
Date: Sun, 07 Feb 2010 18:32:32 -0800
Subject: [C++-sig] Checking for or null PyObject* or PyNone
In-Reply-To: <1265539628.2404.16.camel@murrayc-x61>
References: <1265539628.2404.16.camel@murrayc-x61>
Message-ID: <1265596352.2393.4.camel@bishop>

On Sun, 2010-02-07 at 11:47 +0100, Murray Cumming wrote:
> Doing this
>   boost::python::object obj = get_some_object();
>   if(obj)
>   {
>      //do something
>   }
> seems to check a bool value inside the underlying PyObject, though I
> guess that could throw an exception if it doesn't contain actually
> contain a bool. Is that right?
> The reference documentation is not very clear about this:
> http://www.boost.org/doc/libs/1_42_0/libs/python/doc/v2/object.html
> 
> 
> I had wrongly guessed that it would check if the underlying PyObject*
> was 0. I did this because I had wrapped a PyObject* in a
> boost::python::object for convenience, without checking for 0 first.
> 

I believe the checking happens typically in boost::python::handle, which
is an intermediate between PyObject* and boost::python::object.  When
you construct a handle, you can pass wrap it with null_ok if you don't
want it to throw on NULL; otherwise I think it would.  How are you
making the boost::python::object?  I always construct a handle first.

> Is there any correct way to check, other than doing
>   if(obj.ptr())
> ?
> 
> 
> And what's the correct way to check for PyNone other than 
>   if (obj == boost::python::object())
> ?
> 

I think that's the standard way.  I suppose

if (obj.ptr() == Py_None)

might be faster on some compilers, but it's almost certainly not worth
worrying about.


Jim Bosch



From seefeld at sympatico.ca  Mon Feb  8 05:14:46 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Sun, 07 Feb 2010 23:14:46 -0500
Subject: [C++-sig] Checking for or null PyObject* or PyNone
In-Reply-To: <1265596352.2393.4.camel@bishop>
References: <1265539628.2404.16.camel@murrayc-x61>
	<1265596352.2393.4.camel@bishop>
Message-ID: <4B6F8FB6.8090702@sympatico.ca>

On 02/07/2010 09:32 PM, Jim Bosch wrote:
> On Sun, 2010-02-07 at 11:47 +0100, Murray Cumming wrote:
>> And what's the correct way to check for PyNone other than
>>    if (obj == boost::python::object())
>> ?
>>
>>      
> I think that's the standard way.  I suppose
>
> if (obj.ptr() == Py_None)
>
> might be faster on some compilers, but it's almost certainly not worth
> worrying about.
>    

if (obj) ...

is definitely the right way to do it.

     Stefan

-- 

       ...ich hab' noch einen Koffer in Berlin...


From rwgk at yahoo.com  Mon Feb  8 08:02:46 2010
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Sun, 7 Feb 2010 23:02:46 -0800 (PST)
Subject: [C++-sig] Checking for or null PyObject* or PyNone
In-Reply-To: <4B6F8FB6.8090702@sympatico.ca>
References: <1265539628.2404.16.camel@murrayc-x61>
	<1265596352.2393.4.camel@bishop> <4B6F8FB6.8090702@sympatico.ca>
Message-ID: <849443.26697.qm@web111414.mail.gq1.yahoo.com>

> if (obj) ...

> 
> is definitely the right way to do it.

Sorry to correct, but it isn't equivalent to "obj is None".
if (obj) uses PyObject_IsTrue(); see
boost/python/object_operators.hpp line 60.

I'll add

  obj.is_none()

and

  obj.is_not_none()

when I get a chance.

Ralf

From seefeld at sympatico.ca  Mon Feb  8 15:46:43 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Mon, 08 Feb 2010 09:46:43 -0500
Subject: [C++-sig] Checking for or null PyObject* or PyNone
In-Reply-To: <849443.26697.qm@web111414.mail.gq1.yahoo.com>
References: <1265539628.2404.16.camel@murrayc-x61>	<1265596352.2393.4.camel@bishop>
	<4B6F8FB6.8090702@sympatico.ca>
	<849443.26697.qm@web111414.mail.gq1.yahoo.com>
Message-ID: <4B7023D3.6020406@sympatico.ca>

On 02/08/2010 02:02 AM, Ralf W. Grosse-Kunstleve wrote:
>> if (obj) ...
>>      
>    
>> is definitely the right way to do it.
>>      
> Sorry to correct, but it isn't equivalent to "obj is None".
> if (obj) uses PyObject_IsTrue(); see
> boost/python/object_operators.hpp line 60.
>    

Oh, thanks for the correction !

I guess PyObject_IsTrue() returns false for None, so it may just have 
worked by coincidence in the past. But yes, there is a need to 
disambiguate between Py_False and Py_None.

> I'll add
>
>    obj.is_none()
>
> and
>
>    obj.is_not_none()
>
> when I get a chance.
>    

Thanks. I don't think there is a need for more than one addition, 
though. I.e., "obj.is_not_none()" is redundant, and could be spelled 
simply "!obj.is_none()". (Or in fact, "obj.is_none()" may just be the 
same as "obj == object()", though the latter is a little wasteful.

Thanks,
         Stefan

-- 

       ...ich hab' noch einen Koffer in Berlin...


From michaelschuitema at googlemail.com  Tue Feb  9 11:41:28 2010
From: michaelschuitema at googlemail.com (Michael Schuitema)
Date: Tue, 9 Feb 2010 10:41:28 +0000
Subject: [C++-sig] pointer arguments
Message-ID: <9fad84d91002090241h59e55f4cj9ce4859d509c23f6@mail.gmail.com>

Hi all,

I am new to boost.python (and relatively new to python) so my question might
be rather simple (or stupid).

I have an abstract C++ base class and a derived class DerivedA which I
exposed to Python. Works beautifully

class Base
{
public:
    virtual double compute(void) = 0;
};

class DerivedA : public Base
{
public:
     virtual double compute(void) { return 5.0; }
};

added my own DerivedB in python
class DerivedB(Base):
    def compute(self):
        return 25.0;

which also works.
>>> d_a = DerivedA()  // my C++ derived class; works fine
>>> d_b = DerivedB()  // my python derived class; works fine
>>> d_a.compute()     // works fine
5.0
>>> d_b.compute()     // works fine
25.0

Now I have added a funtion testBase in C++ which looks as follows
double testBase(Base* b)
{
    return b->compute() + 11.5;
}

which I exposed to python using (with help of Py++)

 typedef double (*testBase_function_type)(Base *);

def("testBase", testBase_function_type( &::testBase ),
       ( arg("b") ) );

What I was hoping to work was

>>> testBase(d_a) // on the derivedA C++ class
Traceback (most recent call last):
ArgumentError: Python argument types in
    hello.testBase(DerivedA)
did not match C++ signature:
    testBase(class Base * b)

does not work. Any help would be appreciated.
My more general aim is to write C++ code in terms of abstract base classes.
>From time to time I need to derive classes in python from the abstract base
classes, but I would still like to be able to use the functions written in
terms of the C++ base classes. How would I need to go about that using
boost.python

Thanks,
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100209/2501258f/attachment.htm>

From wingusr at gmail.com  Wed Feb 10 10:49:03 2010
From: wingusr at gmail.com (TP)
Date: Wed, 10 Feb 2010 01:49:03 -0800
Subject: [C++-sig] [Boost.Python] tutorial python extension fails on
	import with ImportError: No module named hello_ext
Message-ID: <fe0b845a1002100149j1e59a2dcr4bcfd1623e554705@mail.gmail.com>

I know this is a little late, but I just started using Boost.Python on
Windows XP, and saw the same exact problems as the original poster so
I thought I should give a solution since no one else has (and I've
seen the same question go unanswered elsewhere).

On 11/10/2009 10:36 PM, Charles Lentz wrote:

> When using boost_1_38_0\libs\python\example\tutorial, I try to import the
> extension that is created (hello_ext) but get the exception, ImportError: No
> module named hello_ext

> I modify Jamroot so that the last few lines where it creates tests are
> commented out.  When I run bjam again, hello_ext.pyd remains (previously it
> was being created, used for the test, and then deleted).  I try to run
> hello.py (which is the script that the test uses) from IDLE (which comes
> with the installation of python) and it throws an exception for my import
> statement

As others note elsewhere you can instead add "--preserve-test-targets"
to your bjam command line. Also adding "-d+2" lets you see more
verbose output of what bjam is doing behind the scenes.

>
> Traceback (most recent call last):
>
>  File
> "C:\Users\Charles\Desktop\boost_1_38_0\boost_1_38_0\libs\python\example\tuto
> rial\hello.py", line 6, in <module>

>    import hello_ext
>
> ImportError: No module named hello_ext
>
> Hrmm.  Okay.  So I move the hello.py file into the same directory as the
> hello_ext.pyd file.  Run it again using IDLE, still get the exception
>

After much googling and looking at the bjam output with "-d+2" turned
on, I discovered that the problem is not just with the location (and
existence) of hello_ext.pyd.

Not only does hello_ext.pyd have to be somewhere in your PATH (copying
it to the same dir as hello.py solves that problem), but the
Boost.Python dll ALSO has to be in the path. In my case I had to add
C:\boost_1_42_0\stage\lib to my PATH.

Of course, this is obvious after the fact. However, the error message
is confusing. What it really means in this case is:

  Can't find boost_python-vc90-mt-1_42.dll or
  boost_python-vc90-mt-gd-1_42.dll.

In addition the "Getting Started on Windows" page at
http://www.boost.org/doc/libs/1_42_0/more/getting_started/windows.html
NEVER mentions having to add C:\boost_1_42_0\stage\lib to the PATH if
you compile the library yourself.

I imagine this doesn't come up more often because most people
initially use the pre-built binaries from BoostPro Computing, and that
installer probably correctly sets the PATH.

I also had problems running bjam since I had copied the tutorial to my
F: drive. I was getting:

  Unable to load Boost.Build: could not find "boost-build.jam"
  ---------------------------------------------------------------
  Attempted search from F:\Boost\BoostPythonTutorial up to the root

Changing the tutorial's jamroot file to say:

  use-project boost
    : C:/boost_1_42_0 ;

didn't help.

I eventually had to copy C:\boost_1_42_0\boost-build.jam to F:\Boost
and comment everything out and add:

  boost-build C:/boost_1_42_0/tools/build/v2 ;

(This last part may not have normally been necessary but I got some
error message about not being able to find a dll when I first ran
bootstrap.bat. I solved that problem by downloading the prebuilt
version of bjam.exe)

Turns out all of this is explained here:

 Knowledge Base/CPP/Boost Python Step By Step
 http://www.thalesians.com/finance/index.php/Knowledge_Base/CPP/Boost_Python_Step_By_Step#Building_the_Boost.Python_Tutorial_in_Another_Location

From murrayc at murrayc.com  Thu Feb 11 13:18:46 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Thu, 11 Feb 2010 13:18:46 +0100
Subject: [C++-sig] Getting object for PyObject (reference counting)
In-Reply-To: <49A874AE.6010306@pixar.com>
References: <1235727190.21945.25.camel@murrayc-desktop>
	<49A8162A.6060505@pixar.com> <1235769922.8034.21.camel@murrayc-x61>
	<49A874AE.6010306@pixar.com>
Message-ID: <1265890726.14276.7.camel@murrayc-desktop>

On Fri, 2009-02-27 at 15:18 -0800, Alex Mohr wrote:
[snip]
>  If you have a PyObject *p and you want a bp::object, 
> construct it via:
> 
> object(handle<>(p))           // when p's a new reference

I find that I have to split the handle and object over two lines. If I
do this
  boost::python::object cppobject(boost::python::handle<>(cObject));
and then something like this:
  if(!cppobject.ptr())
    doSomething()

then I get this weird compiler error on the if() line:

glom/python_embed/glom_python.cc:229: error: request for member ?ptr? in
?cppobject?, which is of non-class type
?boost::python::api::object(boost::python::handle<_object>)?

It's as if doing it on one line has changed the type. Maybe I've
discovered some weird bug with g++ 4.4

> object(handle<>(borrowed(p))) // when p's a borrowed reference

However, this works fine, when it's what I want.

-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From seefeld at sympatico.ca  Thu Feb 11 13:57:27 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Thu, 11 Feb 2010 07:57:27 -0500
Subject: [C++-sig] Getting object for PyObject (reference counting)
In-Reply-To: <1265890726.14276.7.camel@murrayc-desktop>
References: <1235727190.21945.25.camel@murrayc-desktop>	<49A8162A.6060505@pixar.com>
	<1235769922.8034.21.camel@murrayc-x61>	<49A874AE.6010306@pixar.com>
	<1265890726.14276.7.camel@murrayc-desktop>
Message-ID: <4B73FEB7.1090108@sympatico.ca>

On 02/11/2010 07:18 AM, Murray Cumming wrote:
> On Fri, 2009-02-27 at 15:18 -0800, Alex Mohr wrote:
> [snip]
>    
>>   If you have a PyObject *p and you want a bp::object,
>> construct it via:
>>
>> object(handle<>(p))           // when p's a new reference
>>      
> I find that I have to split the handle and object over two lines. If I
> do this
>    boost::python::object cppobject(boost::python::handle<>(cObject));
> and then something like this:
>    if(!cppobject.ptr())
>      doSomething()
>
> then I get this weird compiler error on the if() line:
>
> glom/python_embed/glom_python.cc:229: error: request for member ?ptr? in
> ?cppobject?, which is of non-class type
> ?boost::python::api::object(boost::python::handle<_object>)?
>
> It's as if doing it on one line has changed the type.

It does !

In C++, if something may be interpreted as a declaration, it is a 
declaration.
And in the above case, the first line may be interpreted as a 
declaration of a function "cppobject" returning a bp::object, taking a 
handle<>.

If you split things into three statements, by first instantiating a 
(named) handle<>, then pass that to the cppobject constructor, the error 
will go away, since that line then is no longer ambiguous.

HTH,
         Stefan


-- 

       ...ich hab' noch einen Koffer in Berlin...


From murrayc at murrayc.com  Thu Feb 11 18:30:41 2010
From: murrayc at murrayc.com (Murray Cumming)
Date: Thu, 11 Feb 2010 18:30:41 +0100
Subject: [C++-sig] boost::python and Date/Time values
In-Reply-To: <1265579960.8678.5.camel@murrayc-x61>
References: <1265579960.8678.5.camel@murrayc-x61>
Message-ID: <1265909441.14276.253.camel@murrayc-desktop>

On Sun, 2010-02-07 at 22:59 +0100, Murray Cumming wrote:
> Is there any easy way to get python Date, Time and DateTime values from
> boost::python::objects? In C, I'm using PyDateTime_GET_YEAR(),
> PyDateTime_GET_MONTH(), PyDateTime_GET_DAY(), etc, but it would be nice
> to use some C++ API for this.

Any ideas?


-- 
murrayc at murrayc.com
www.murrayc.com
www.openismus.com


From troy at resophonic.com  Thu Feb 11 18:55:44 2010
From: troy at resophonic.com (troy d. straszheim)
Date: Thu, 11 Feb 2010 12:55:44 -0500
Subject: [C++-sig] boost::python and Date/Time values
In-Reply-To: <1265909441.14276.253.camel@murrayc-desktop>
References: <1265579960.8678.5.camel@murrayc-x61>
	<1265909441.14276.253.camel@murrayc-desktop>
Message-ID: <4B7444A0.7080107@resophonic.com>

Murray Cumming wrote:
> On Sun, 2010-02-07 at 22:59 +0100, Murray Cumming wrote:
>> Is there any easy way to get python Date, Time and DateTime values from
>> boost::python::objects? In C, I'm using PyDateTime_GET_YEAR(),
>> PyDateTime_GET_MONTH(), PyDateTime_GET_DAY(), etc, but it would be nice
>> to use some C++ API for this.
> 
> Any ideas?
> 

Last time I had to do this it was for an app that had a custom date/time 
class on the c++ side, we rolled our own converters using the 
PyDateTime_*() methods.  Some standard converters for boost::date_time 
classes would be nice...

-t





From jvogel at vtiinstruments.com  Thu Feb 11 19:16:22 2010
From: jvogel at vtiinstruments.com (James Vogel)
Date: Thu, 11 Feb 2010 12:16:22 -0600
Subject: [C++-sig]  Exceptions with pybindgen and gccxmlparser
Message-ID: <EDAC3201DE40C348ACC19322CA14EB9A0569EF6CA9@34093-MBX-C05.mex07a.mlsrvr.com>

Hello --

Trying to use pybindgen.gccxmlparser, and running into an issue with exception handling.

I have a virtual class that declares
virtual int throw_error() throw (std::exception);
and a non-virtual class that extends the virtual class and overrides throw_an_error to throw an actual error. I run the module parser and generate foo.cpp. When I build foo.cpp, it has a build error:
"looser throw specifier for 'virtual int PyNon_virtual_class__PythonHelper::throw_error()' overriding 'virtual int non_virtual_class::throw_error() throw (std::exception)"

If I remove the function declaration from the virtual class (so that the implementation is not overriding an inherited method), it works correctly. Similarly, if I go into foo.cpp and manually add 'throw (std::exception)' to the relevant declarations, it works, so I suspect there's something afoot in the gccxml parsing that isn't picking up the throw clause in the virtual class; I'm just not sure if there's an annotation I might be missing, or some other issue.

Anyone seen this before?

James.

From rwgk at yahoo.com  Thu Feb 11 20:18:35 2010
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Thu, 11 Feb 2010 11:18:35 -0800 (PST)
Subject: [C++-sig] Getting object for PyObject (reference counting)
In-Reply-To: <4B73FEB7.1090108@sympatico.ca>
References: <1235727190.21945.25.camel@murrayc-desktop>
	<49A8162A.6060505@pixar.com> <1235769922.8034.21.camel@murrayc-x61>
	<49A874AE.6010306@pixar.com>
	<1265890726.14276.7.camel@murrayc-desktop>
	<4B73FEB7.1090108@sympatico.ca>
Message-ID: <872822.99120.qm@web111408.mail.gq1.yahoo.com>

>    boost::python::object cppobject(boost::python::handle<>(cObject));


It should work like this:

boost::python::object cppobject((boost::python::handle<>(cObject)));

(Not that I ever understood exactly why... (or even wanted to understand))

From gjcarneiro at gmail.com  Fri Feb 12 15:25:51 2010
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Fri, 12 Feb 2010 14:25:51 +0000
Subject: [C++-sig] Exceptions with pybindgen and gccxmlparser
In-Reply-To: <EDAC3201DE40C348ACC19322CA14EB9A0569EF6CA9@34093-MBX-C05.mex07a.mlsrvr.com>
References: <EDAC3201DE40C348ACC19322CA14EB9A0569EF6CA9@34093-MBX-C05.mex07a.mlsrvr.com>
Message-ID: <a467ca4f1002120625g31885f77o110b8b041a615190@mail.gmail.com>

On Thu, Feb 11, 2010 at 6:16 PM, James Vogel <jvogel at vtiinstruments.com>wrote:

> Hello --
>
> Trying to use pybindgen.gccxmlparser, and running into an issue with
> exception handling.
>
> I have a virtual class that declares
> virtual int throw_error() throw (std::exception);
> and a non-virtual class that extends the virtual class and overrides
> throw_an_error to throw an actual error. I run the module parser and
> generate foo.cpp. When I build foo.cpp, it has a build error:
> "looser throw specifier for 'virtual int
> PyNon_virtual_class__PythonHelper::throw_error()' overriding 'virtual int
> non_virtual_class::throw_error() throw (std::exception)"
>
> If I remove the function declaration from the virtual class (so that the
> implementation is not overriding an inherited method), it works correctly.
> Similarly, if I go into foo.cpp and manually add 'throw (std::exception)' to
> the relevant declarations, it works, so I suspect there's something afoot in
> the gccxml parsing that isn't picking up the throw clause in the virtual
> class; I'm just not sure if there's an annotation I might be missing, or
> some other issue.
>
> Anyone seen this before?
>

I have not seen it because I have never tested it.  The generated helper
class is missing the "throw (exception)" in the declaration of the
overridden virtual method.  Sounds like a bug -->
https://bugs.launchpad.net/pybindgen/+filebug

Although, please note that translation of Python exceptions into C++ is not
implemented yet, only in the C++ to Python direction.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, UTM, WiN, http://win.inescporto.pt/gjc
"The universe is always one step beyond logic." -- Frank Herbert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100212/afd147b7/attachment.htm>

From peoro.noob at gmail.com  Sat Feb 13 16:41:35 2010
From: peoro.noob at gmail.com (peoro)
Date: Sat, 13 Feb 2010 16:41:35 +0100
Subject: [C++-sig] [Py++] Indexing suite 2: missing methods
Message-ID: <1c7ed23c1002130741y7488345cvd0cffb651634531f@mail.gmail.com>

Hello,
I noticed that some Py++ traits for standard containers are missing a
few methods that I think would be needed.
For example `set_traits' (container traits for `std::set') is missing
`method_len' that,  in my opinion, should be provided.
Is this a feature or a bug? And in the former case, how would I be
supposed to get a set's size? explicitly exposing a `size' function
for it?

From roman.yakovenko at gmail.com  Sat Feb 13 20:05:57 2010
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Sat, 13 Feb 2010 21:05:57 +0200
Subject: [C++-sig] [Py++] Indexing suite 2: missing methods
In-Reply-To: <1c7ed23c1002130741y7488345cvd0cffb651634531f@mail.gmail.com>
References: <1c7ed23c1002130741y7488345cvd0cffb651634531f@mail.gmail.com>
Message-ID: <7465b6171002131105q38d44e3v82d2a046a4d52fbd@mail.gmail.com>

On Sat, Feb 13, 2010 at 5:41 PM, peoro <peoro.noob at gmail.com> wrote:
> Hello,
> I noticed that some Py++ traits for standard containers are missing a
> few methods that I think would be needed.
> For example `set_traits' (container traits for `std::set') is missing
> `method_len' that, ?in my opinion, should be provided.

It is provided.

> Is this a feature or a bug? And in the former case, how would I be
> supposed to get a set's size? explicitly exposing a `size' function
> for it?

Did you try len( x )?


-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

From peoro.noob at gmail.com  Sun Feb 14 17:22:18 2010
From: peoro.noob at gmail.com (peoro)
Date: Sun, 14 Feb 2010 17:22:18 +0100
Subject: [C++-sig] [Py++] Indexing suite 2: missing methods
In-Reply-To: <7465b6171002131105q38d44e3v82d2a046a4d52fbd@mail.gmail.com>
References: <1c7ed23c1002130741y7488345cvd0cffb651634531f@mail.gmail.com>
	<7465b6171002131105q38d44e3v82d2a046a4d52fbd@mail.gmail.com>
Message-ID: <1c7ed23c1002140822j7b142c9cq839b86b3a1721063@mail.gmail.com>

On Sat, Feb 13, 2010 at 8:05 PM, Roman Yakovenko
<roman.yakovenko at gmail.com> wrote:
> On Sat, Feb 13, 2010 at 5:41 PM, peoro <peoro.noob at gmail.com> wrote:
>> Hello,
>> I noticed that some Py++ traits for standard containers are missing a
>> few methods that I think would be needed.
>> For example `set_traits' (container traits for `std::set') is missing
>> `method_len' that, ?in my opinion, should be provided.
>
> It is provided.
>
>> Is this a feature or a bug? And in the former case, how would I be
>> supposed to get a set's size? explicitly exposing a `size' function
>> for it?
>
> Did you try len( x )?
>

Yes, I tried and it isn't working.

This is the C++ code I'm trying to expose using Py++:
#include <set>
std::set<int> f( ) {
	return std::set<int>();
}

Then, in python:
>>> from set_len_test import *
>>> x = f()
>>> len( x )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'set_less__int__greater_' has no len()

If I manually add `method_len' to `supported_methods' in
`indexing_suite/set.hpp', and compile again the module, everything
works as it should:
>>> from set_len_len import *
>>> x = f()
>>> len( x )
0

I've been trying with SVN revisions 1814 and 1824.

>
> --
> Roman Yakovenko
> C++ Python language binding
> http://www.language-binding.net/
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

From roman.yakovenko at gmail.com  Sun Feb 14 20:39:25 2010
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Sun, 14 Feb 2010 21:39:25 +0200
Subject: [C++-sig] [Py++] Indexing suite 2: missing methods
In-Reply-To: <1c7ed23c1002140822j7b142c9cq839b86b3a1721063@mail.gmail.com>
References: <1c7ed23c1002130741y7488345cvd0cffb651634531f@mail.gmail.com>
	<7465b6171002131105q38d44e3v82d2a046a4d52fbd@mail.gmail.com>
	<1c7ed23c1002140822j7b142c9cq839b86b3a1721063@mail.gmail.com>
Message-ID: <7465b6171002141139q421d5a81o94c83627a1beb2a1@mail.gmail.com>

On Sun, Feb 14, 2010 at 6:22 PM, peoro <peoro.noob at gmail.com> wrote:
> On Sat, Feb 13, 2010 at 8:05 PM, Roman Yakovenko
> <roman.yakovenko at gmail.com> wrote:
>> Did you try len( x )?
>>
>
> Yes, I tried and it isn't working.
> ...
> I've been trying with SVN revisions 1814 and 1824.

Can you create small and complete example? ( C++ code, Py++ script and
Py++ generated code )?

Thanks

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

From wingusr at gmail.com  Mon Feb 15 01:32:42 2010
From: wingusr at gmail.com (TP)
Date: Sun, 14 Feb 2010 16:32:42 -0800
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
Message-ID: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>

Suppose I am attempting to use a C (not C++) library that has an .h file
with things like this:

 extern PIX * pixCreate ( l_int32 width, l_int32 height, l_int32 depth );
 extern void pixDestroy ( PIX **ppix );

 extern PIX * pixClone ( PIX *pixs );
 extern PIX * pixCopy ( PIX *pixd, PIX *pixs );

Is it possible to use Boost.Python to wrap such functions?

In particular, note that pixDestroy() is going to want the address of the
pointer returned by pixCreate().

(This is from the leptprotos.h file of the Leptonica Image Processing
library at leptonica.com)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100214/4efb2968/attachment.htm>

From charlessolar at gmail.com  Mon Feb 15 01:43:52 2010
From: charlessolar at gmail.com (Charles Solar)
Date: Sun, 14 Feb 2010 18:43:52 -0600
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
Message-ID: <708326ee1002141643n3d84f151q344bc1c304ec6203@mail.gmail.com>

If you want to wrap the library in C I believe you will have to use the
standard python API, as boost requires C++ features.
It sounds like you do not want to write a wrapper library and instead add
code to leptonica to make it python import-able.  I would not suggest this
approach.

If you want to write a wrapper around the C library in C++ you can use boost
python, then in python you would import your wrapper which would allow you
to use the C library in python using your C++ wrapper functions.

If you just want to add code to this library and make it work in python
immediately check out SWIG, although I am not sure if swig produces cpython
code or boost python C++ code.

On Sun, Feb 14, 2010 at 6:32 PM, TP <wingusr at gmail.com> wrote:

> Suppose I am attempting to use a C (not C++) library that has an .h file
> with things like this:
>
>  extern PIX * pixCreate ( l_int32 width, l_int32 height, l_int32 depth );
>  extern void pixDestroy ( PIX **ppix );
>
>  extern PIX * pixClone ( PIX *pixs );
>  extern PIX * pixCopy ( PIX *pixd, PIX *pixs );
>
> Is it possible to use Boost.Python to wrap such functions?
>
> In particular, note that pixDestroy() is going to want the address of the
> pointer returned by pixCreate().
>
> (This is from the leptprotos.h file of the Leptonica Image Processing
> library at leptonica.com)
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100214/1e03ce0f/attachment.htm>

From seefeld at sympatico.ca  Mon Feb 15 02:04:36 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Sun, 14 Feb 2010 20:04:36 -0500
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
Message-ID: <4B789DA4.1010102@sympatico.ca>

On 02/14/2010 07:32 PM, TP wrote:
> Suppose I am attempting to use a C (not C++) library that has an .h 
> file with things like this:
>
>  extern PIX * pixCreate ( l_int32 width, l_int32 height, l_int32 depth );
>  extern void pixDestroy ( PIX **ppix );
>
>  extern PIX * pixClone ( PIX *pixs );
>  extern PIX * pixCopy ( PIX *pixd, PIX *pixs );
>
> Is it possible to use Boost.Python to wrap such functions?

Yes. Do you have the definition of PIX available ? In that case, you 
could use that via class_<PIX>....
Otherwise, you may want to create a tiny wrapper class for it, such as

class PIXWrapper
{
public:
   PIXWrapper(I_int32 w, I_int32 h, I_int32 d) : pix_(pixCreate(w, h, d) {}
  ~PIXWrapper() { pixDestroy(&pix_);}
  ...
private:
   PIX *pix_;
};

and then use that in the way described in the docs.

HTH,
         Stefan

-- 

       ...ich hab' noch einen Koffer in Berlin...


From roman.yakovenko at gmail.com  Mon Feb 15 06:31:43 2010
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Mon, 15 Feb 2010 07:31:43 +0200
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
Message-ID: <7465b6171002142131y54444550j848043d75f54e39c@mail.gmail.com>

On Mon, Feb 15, 2010 at 2:32 AM, TP <wingusr at gmail.com> wrote:
> Suppose I am attempting to use a C (not C++) library that has an .h file
> with things like this:
>
> ?extern PIX * pixCreate ( l_int32 width, l_int32 height, l_int32 depth );
> ?extern void pixDestroy ( PIX **ppix );
>
> ?extern PIX * pixClone ( PIX *pixs );
> ?extern PIX * pixCopy ( PIX *pixd, PIX *pixs );
>
> Is it possible to use Boost.Python to wrap such functions?

Definitely, but the price will be too high.

If the library coded in "C" only than you can use ctypes module to
access the functionality. You will get pure-Python solution.
There are even few code generators out there, that will happily
generate initial ctypes code for you.

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

From peoro.noob at gmail.com  Mon Feb 15 12:30:45 2010
From: peoro.noob at gmail.com (peoro)
Date: Mon, 15 Feb 2010 12:30:45 +0100
Subject: [C++-sig] [Py++] Indexing suite 2: missing methods
In-Reply-To: <7465b6171002141139q421d5a81o94c83627a1beb2a1@mail.gmail.com>
References: <1c7ed23c1002130741y7488345cvd0cffb651634531f@mail.gmail.com>
	<7465b6171002131105q38d44e3v82d2a046a4d52fbd@mail.gmail.com>
	<1c7ed23c1002140822j7b142c9cq839b86b3a1721063@mail.gmail.com>
	<7465b6171002141139q421d5a81o94c83627a1beb2a1@mail.gmail.com>
Message-ID: <1c7ed23c1002150330q2a5cec44wfbf6ce238f577f52@mail.gmail.com>

On Sun, Feb 14, 2010 at 8:39 PM, Roman Yakovenko
<roman.yakovenko at gmail.com> wrote:
> On Sun, Feb 14, 2010 at 6:22 PM, peoro <peoro.noob at gmail.com> wrote:
>> On Sat, Feb 13, 2010 at 8:05 PM, Roman Yakovenko
>> <roman.yakovenko at gmail.com> wrote:
>>> Did you try len( x )?
>>>
>>
>> Yes, I tried and it isn't working.
>> ...
>> I've been trying with SVN revisions 1814 and 1824.
>
> Can you create small and complete example? ( C++ code, Py++ script and
> Py++ generated code )?
>
> Thanks
>

Sure:

##### main.cpp
#include <set>
std::set<int> f( ) {
        return std::set<int>();
}

##### export.py
import os
from pyplusplus import module_builder
here = os.path.abspath( r"." )
mb = module_builder.module_builder_t( [here + r"/main.cpp"],
working_directory=here, indexing_suite_version=2 )
mb.build_code_creator( module_name='set_test_len' )
mb.write_module( here + r'/bindings.cpp' )

##### set_test_len.py
from set_test_len import *
x = f()
len( x )

$ python export.py
INFO Parsing source file
"/net/pacifico/l/disc1/home/giangran/C/boost/python/set_len_test/main.cpp"
...
INFO gccxml cmd:
/usr/bin/gccxml-I"/net/pacifico/l/disc1/home/giangran/C/boost/python/set_len_test"
"/net/pacifico/l/disc1/home/giangran/C/boost/python/set_len_test/main.cpp"
-fxml="/tmp/tmpb3p-BN.xml"
INFO GCCXML version - 0.9( 1.128 )
INFO: file "named_tuple.py" - updated( 0.000000 seconds )
INFO: file "set.hpp" - updated( 0.000000 seconds )
INFO: file "slice.hpp" - updated( 0.000000 seconds )
INFO: file "element_proxy_traits.hpp" - updated( 0.000000 seconds )
INFO: file "python_iterator.hpp" - updated( 0.010000 seconds )
INFO: file "proxy_iterator.hpp" - updated( 0.000000 seconds )
INFO: file "element_proxy.hpp" - updated( 0.010000 seconds )
INFO: file "container_suite.hpp" - updated( 0.010000 seconds )
INFO: file "slice_handler.hpp" - updated( 0.000000 seconds )
INFO: file "workaround.hpp" - updated( 0.000000 seconds )
INFO: file "value_traits.hpp" - updated( 0.000000 seconds )
INFO: file "visitor.hpp" - updated( 0.000000 seconds )
INFO: file "algorithms.hpp" - updated( 0.000000 seconds )
INFO: file "vector.hpp" - updated( 0.000000 seconds )
INFO: file "methods.hpp" - updated( 0.000000 seconds )
INFO: file "deque.hpp" - updated( 0.000000 seconds )
INFO: file "shared_proxy_impl.hpp" - updated( 0.000000 seconds )
INFO: file "iterator_range.hpp" - updated( 0.000000 seconds )
INFO: file "int_slice_helper.hpp" - updated( 0.000000 seconds )
INFO: file "container_traits.hpp" - updated( 0.000000 seconds )
INFO: file "suite_utils.hpp" - updated( 0.000000 seconds )
INFO: file "list.hpp" - updated( 0.000000 seconds )
INFO: file "map.hpp" - updated( 0.000000 seconds )
INFO: file "iterator_traits.hpp" - updated( 0.000000 seconds )
INFO: file "container_proxy.hpp" - updated( 0.000000 seconds )
INFO: file "multimap.hpp" - updated( 0.000000 seconds )
INFO: file "pair.hpp" - updated( 0.000000 seconds )
INFO: file "registry_utils.hpp" - updated( 0.000000 seconds )
INFO: file "bindings.cpp" - updated( 0.000000 seconds )

$ g++ `python-config --cflags` -lboost_python -shared -fPIC
bindings.cpp -o set_test_len.so
cc1plus: warning: command line option "-Wstrict-prototypes" is valid
for Ada/C/ObjC but not for C++

$ python set_test_len.py
Traceback (most recent call last):
  File "set_test_len.py", line 3, in <module>
    len( x )
TypeError: object of type 'set_less__int__greater_' has no len()

> --
> Roman Yakovenko
> C++ Python language binding
> http://www.language-binding.net/
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

From roman.yakovenko at gmail.com  Mon Feb 15 21:28:22 2010
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Mon, 15 Feb 2010 22:28:22 +0200
Subject: [C++-sig] [Py++] Indexing suite 2: missing methods
In-Reply-To: <1c7ed23c1002150330q2a5cec44wfbf6ce238f577f52@mail.gmail.com>
References: <1c7ed23c1002130741y7488345cvd0cffb651634531f@mail.gmail.com>
	<7465b6171002131105q38d44e3v82d2a046a4d52fbd@mail.gmail.com>
	<1c7ed23c1002140822j7b142c9cq839b86b3a1721063@mail.gmail.com>
	<7465b6171002141139q421d5a81o94c83627a1beb2a1@mail.gmail.com>
	<1c7ed23c1002150330q2a5cec44wfbf6ce238f577f52@mail.gmail.com>
Message-ID: <7465b6171002151228u5a5b138bi16a7c47f6ebba24d@mail.gmail.com>

On Mon, Feb 15, 2010 at 1:30 PM, peoro <peoro.noob at gmail.com> wrote:
> On Sun, Feb 14, 2010 at 8:39 PM, Roman Yakovenko
>> Can you create small and complete example? ( C++ code, Py++ script and
>> Py++ generated code )?
>>
>> Thanks
>>
>
> Sure:
>
> ##### main.cpp
> #include <set>
> std::set<int> f( ) {
> ? ? ? ?return std::set<int>();
> }
>
> ##### export.py
> import os
> from pyplusplus import module_builder
> here = os.path.abspath( r"." )
> mb = module_builder.module_builder_t( [here + r"/main.cpp"],
> working_directory=here, indexing_suite_version=2 )
> mb.build_code_creator( module_name='set_test_len' )
> mb.write_module( here + r'/bindings.cpp' )
>
> ##### set_test_len.py
> from set_test_len import *
> x = f()
> len( x )
>

Look, I am not sure what is going wrong. I verified the example you
gave and it works as expected(
http://pygccxml.svn.sourceforge.net/pygccxml/?rev=1825&view=rev )

The functionality you are talking about is pretty fundamental and used
in other projects too.

The following is the code generated by Py++ to expose std::set<int>:
    { //::std::set< int >
        typedef bp::class_< std::set< int > > set_less__int__greater__exposer_t;
        set_less__int__greater__exposer_t
set_less__int__greater__exposer = set_less__int__greater__exposer_t(
"set_less__int__greater_", "documentation" );
        bp::scope set_less__int__greater__scope(
set_less__int__greater__exposer );
        set_less__int__greater__exposer.def( bp::indexing::set_suite<
std::set< int > >() );
    }

Does your generated code look the same?

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

From wingusr at gmail.com  Mon Feb 15 22:26:00 2010
From: wingusr at gmail.com (TP)
Date: Mon, 15 Feb 2010 13:26:00 -0800
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <4B789DA4.1010102@sympatico.ca>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
	<4B789DA4.1010102@sympatico.ca>
Message-ID: <fe0b845a1002151326kfdc0b15v3d84c7097130a5ae@mail.gmail.com>

On Sun, Feb 14, 2010 at 5:04 PM, Stefan Seefeld <seefeld at sympatico.ca> wrote:
> Yes. Do you have the definition of PIX available ? In that case, you could
> use that via class_<PIX>....

Yes. The entire source of the Leptonica C Image Processing Library is
available at http::/leptonica.com.

Could you give a little more detail please?

The part that confuses me at this point is how to handle parameters of
the form PIX** ppix (Leptonica is FULL of parameters like SOMETYPE**
someParm).

> Otherwise, you may want to create a tiny wrapper class for it, such as
>
> class PIXWrapper
> {
> public:
>  PIXWrapper(I_int32 w, I_int32 h, I_int32 d) : pix_(pixCreate(w, h, d) {}
>  ~PIXWrapper() { pixDestroy(&pix_);}
>  ...
> private:
>  PIX *pix_;
> };

I was hoping to not have to write such wrapper classes. But thank you
for the example. From that I can clearly see how to handle the address
of pointer problem if I decide to take this route.

From seefeld at sympatico.ca  Mon Feb 15 22:41:17 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Mon, 15 Feb 2010 16:41:17 -0500
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <fe0b845a1002151326kfdc0b15v3d84c7097130a5ae@mail.gmail.com>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>	<4B789DA4.1010102@sympatico.ca>
	<fe0b845a1002151326kfdc0b15v3d84c7097130a5ae@mail.gmail.com>
Message-ID: <4B79BF7D.1030604@sympatico.ca>

On 02/15/2010 04:26 PM, TP wrote:
> On Sun, Feb 14, 2010 at 5:04 PM, Stefan Seefeld<seefeld at sympatico.ca>  wrote:
>    
>> Yes. Do you have the definition of PIX available ? In that case, you could
>> use that via class_<PIX>....
>>      
> Yes. The entire source of the Leptonica C Image Processing Library is
> available at http::/leptonica.com.
>
> Could you give a little more detail please?
>    

If PIX is a known type (as opposed to an opaque pointer), you can wrap 
that directly, instead of having to use an indirection.
However, you still need to find a way to specify how PIX objects are to 
be constructed and destroyed, and thus, you need to store PIX inside 
Python objects via pointers that allow custom destructors (such as 
boost::shared_ptr).

I bet, all things considered, a wrapper class such as the one I proposed 
in my previous mail is still the best way forward, as it best 
encapsulates how PIX objects get constructed (incl. cloned) and destructed.

Memory management is by far the most complex aspect of wrapping C and 
C++ libraries in Python. And if your library mostly operates on raw 
pointers, you really need to be careful about object lifetime, argument 
and return value ownership, etc.


Regards,
         Stefan

-- 

       ...ich hab' noch einen Koffer in Berlin...


From wingusr at gmail.com  Mon Feb 15 22:44:34 2010
From: wingusr at gmail.com (TP)
Date: Mon, 15 Feb 2010 13:44:34 -0800
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <708326ee1002141643n3d84f151q344bc1c304ec6203@mail.gmail.com>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
	<708326ee1002141643n3d84f151q344bc1c304ec6203@mail.gmail.com>
Message-ID: <fe0b845a1002151344s4ea3748boa56736e936364110@mail.gmail.com>

Doesn't the first example in The Boost.Python Tutorial
(http://www.boost.org/doc/libs/1_42_0/libs/python/doc/tutorial/doc/html/index.html)
show it making a simple C function available to Python?

I want to be able to use the open source, Leptonica C Image Processing
Library (available at http::/leptonica.com) from Python, preferably
without having to make any changes to it. But again, handling the
PIX** parameter seems problematic.

I was hoping to avoid writing C++ wrapper classes (but Stefan
Seefeld's answer on this list) does have a nice short example of what
I would have to do.

Using SWIG was my last choice (well, the last choice is directly using
the C Api).

I have also looked at pybindgen, but Gustavo Carneiro tells me at
https://answers.launchpad.net/pybindgen/+question/101056 that it is
currently unable to easily handle parameters of the form SOMETYPE**.

Other options I will investigate are: ctypes, Cython, expy, and SIP
(the last only because the ultiate goal is to use Leptonica Image
Processing routines inside a PyQt-based app).

From wingusr at gmail.com  Mon Feb 15 22:50:17 2010
From: wingusr at gmail.com (TP)
Date: Mon, 15 Feb 2010 13:50:17 -0800
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <7465b6171002142131y54444550j848043d75f54e39c@mail.gmail.com>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
	<7465b6171002142131y54444550j848043d75f54e39c@mail.gmail.com>
Message-ID: <fe0b845a1002151350r2cad9286we8763a92bc122cbc@mail.gmail.com>

On Sun, Feb 14, 2010 at 9:31 PM, Roman Yakovenko
<roman.yakovenko at gmail.com> wrote:
> On Mon, Feb 15, 2010 at 2:32 AM, TP <wingusr at gmail.com> wrote:
>> Suppose I am attempting to use a C (not C++) library that has an .h file
>> with things like this:
>>
>> ?extern PIX * pixCreate ( l_int32 width, l_int32 height, l_int32 depth );
>> ?extern void pixDestroy ( PIX **ppix );
>>
>> ?extern PIX * pixClone ( PIX *pixs );
>> ?extern PIX * pixCopy ( PIX *pixd, PIX *pixs );
>>
>> Is it possible to use Boost.Python to wrap such functions?
>
> Definitely, but the price will be too high.
>
> If the library coded in "C" only than you can use ctypes module to
> access the functionality. You will get pure-Python solution.
> There are even few code generators out there, that will happily
> generate initial ctypes code for you.
>
> --
> Roman Yakovenko
> C++ Python language binding
> http://www.language-binding.net/
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

Thank you for your response. ctypes was on my list of options to
investigate. I wonder if you know if it will be able to handle
parameters like PIX** ppix?

A generator would be nice, as the Leptonica Image Processing Library
is quite large (although my initial plans are to do as few functions
as possible, just to see what's involved in making this work).

From wingusr at gmail.com  Mon Feb 15 23:18:48 2010
From: wingusr at gmail.com (TP)
Date: Mon, 15 Feb 2010 14:18:48 -0800
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <4B79BF7D.1030604@sympatico.ca>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
	<4B789DA4.1010102@sympatico.ca>
	<fe0b845a1002151326kfdc0b15v3d84c7097130a5ae@mail.gmail.com>
	<4B79BF7D.1030604@sympatico.ca>
Message-ID: <fe0b845a1002151418o40504831ge791cc4b80c1c753@mail.gmail.com>

On Mon, Feb 15, 2010 at 1:41 PM, Stefan Seefeld <seefeld at sympatico.ca> wrote:
> On 02/15/2010 04:26 PM, TP wrote:
>>
>> On Sun, Feb 14, 2010 at 5:04 PM, Stefan Seefeld<seefeld at sympatico.ca>
>>  wrote:
>>
>>>
>>> Yes. Do you have the definition of PIX available ? In that case, you
>>> could
>>> use that via class_<PIX>....
>>>
>>
>> Yes. The entire source of the Leptonica C Image Processing Library is
>> available at http::/leptonica.com.
>>
>> Could you give a little more detail please?
>>
>
> If PIX is a known type (as opposed to an opaque pointer), you can wrap that
> directly, instead of having to use an indirection.
> However, you still need to find a way to specify how PIX objects are to be
> constructed and destroyed, and thus, you need to store PIX inside Python
> objects via pointers that allow custom destructors (such as
> boost::shared_ptr).
>
> I bet, all things considered, a wrapper class such as the one I proposed in
> my previous mail is still the best way forward, as it best encapsulates how
> PIX objects get constructed (incl. cloned) and destructed.
>
> Memory management is by far the most complex aspect of wrapping C and C++
> libraries in Python. And if your library mostly operates on raw pointers,
> you really need to be careful about object lifetime, argument and return
> value ownership, etc.
>
>
> Regards,
>        Stefan
>
> --
>
>      ...ich hab' noch einen Koffer in Berlin...
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>

While the definition of a PIX is available, Leptonica users are
supposed to use the supplied functions to access "attributes" (things
like pixGetWidth(), pixSetWidth(), etc). See
http://leptonica.com/library-notes.html#PIX for brief details.

Since images can be large, Leptonica goes to great lengths to avoid
copying them if possible so memory management is crucial. The PIX are
reference counted, and it is up to the caller to make sure they use
pixDestroy() at the right times.

>From pix1.c:

(2) The protocol to be used is:
    (a) Whenever you want a new handle to an existing image,
        call pixClone(), which just bumps a ref count.
    (b) Always call pixDestroy() on all handles.  This
        decrements the ref count, nulls the handle, and
        only destroys the pix when pixDestroy() has been
        called on all handles.

I probably should have picked a simpler library for my first foray
into hooking C/C++ to Python :)

I'm hoping I can initially do small pieces at a time as I need them.

From roman.yakovenko at gmail.com  Mon Feb 15 23:33:02 2010
From: roman.yakovenko at gmail.com (Roman Yakovenko)
Date: Tue, 16 Feb 2010 00:33:02 +0200
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <fe0b845a1002151350r2cad9286we8763a92bc122cbc@mail.gmail.com>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
	<7465b6171002142131y54444550j848043d75f54e39c@mail.gmail.com>
	<fe0b845a1002151350r2cad9286we8763a92bc122cbc@mail.gmail.com>
Message-ID: <7465b6171002151433q60c7388asebaa0592fafc9e1@mail.gmail.com>

On Mon, Feb 15, 2010 at 11:50 PM, TP <wingusr at gmail.com> wrote:
> Thank you for your response. ctypes was on my list of options to
> investigate. I wonder if you know if it will be able to handle
> parameters like PIX** ppix?

Definitely

> A generator would be nice, as the Leptonica Image Processing Library
> is quite large (although my initial plans are to do as few functions
> as possible, just to see what's involved in making this work).

There are few code generators for ctypes module. I can recommend my own one :-).

Take a look on example which exposes gmplib:
http://pygccxml.svn.sourceforge.net/viewvc/pygccxml/pyplusplus_dev/examples/gmplib_dev/

HTH

-- 
Roman Yakovenko
C++ Python language binding
http://www.language-binding.net/

From jooprenes at tele2.nl  Tue Feb 16 00:01:55 2010
From: jooprenes at tele2.nl (joop renes)
Date: Tue, 16 Feb 2010 00:01:55 +0100
Subject: [C++-sig] [Boost.Python] Wrapping C library functions
In-Reply-To: <fe0b845a1002151344s4ea3748boa56736e936364110@mail.gmail.com>
References: <fe0b845a1002141632v4cbb1590i83e5238ccb02f762@mail.gmail.com>
	<708326ee1002141643n3d84f151q344bc1c304ec6203@mail.gmail.com>
	<fe0b845a1002151344s4ea3748boa56736e936364110@mail.gmail.com>
Message-ID: <1266274915.5873.10.camel@joop-desktop>

On Mon, 2010-02-15 at 13:44 -0800, TP wrote:
> Doesn't the first example in The Boost.Python Tutorial
> (http://www.boost.org/doc/libs/1_42_0/libs/python/doc/tutorial/doc/html/index.html)
> show it making a simple C function available to Python?
> 
> I want to be able to use the open source, Leptonica C Image Processing
> Library (available at http::/leptonica.com) from Python, preferably
> without having to make any changes to it. But again, handling the
> PIX** parameter seems problematic.
> 
> I was hoping to avoid writing C++ wrapper classes (but Stefan
> Seefeld's answer on this list) does have a nice short example of what
> I would have to do.
> 
> Using SWIG was my last choice (well, the last choice is directly using
> the C Api).
> 
> I have also looked at pybindgen, but Gustavo Carneiro tells me at
> https://answers.launchpad.net/pybindgen/+question/101056 that it is
> currently unable to easily handle parameters of the form SOMETYPE**.
> 
> Other options I will investigate are: ctypes, Cython, expy, and SIP
> (the last only because the ultiate goal is to use Leptonica Image
> Processing routines inside a PyQt-based app).
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
hi,
i am new to this list, but i have been exposed to c++ with some
intensity and have benefitted greatly from the loki library provided by
Andrei Alexandrescu, author of Modern C++ Design. His class template
SmartPtr provides everything you'd ever want in a wrapper class, memory
management included.Also the library is completely free
<loki-lib.SF>
I hope this helps, because i plan to integrate some python stuff with c
++
best regards
joop



From still.horse at gmail.com  Wed Feb 17 03:31:11 2010
From: still.horse at gmail.com (Kevin Daley)
Date: Tue, 16 Feb 2010 21:31:11 -0500
Subject: [C++-sig] Hi, new around these parts + small link error
Message-ID: <1d80564a1002161831x18479a90r6575257584ce04e1@mail.gmail.com>

Hi, I'm new to these lists, but I'm an experienced developer.

Right now, I'm working on a project using a popular game engine.  I've got
some Boost::Python code that's generating a bit of a link-time error; I was
under the impression that constitutes most of the traffic around here.
 Using MSVC 9 and v.1_36 of the library (not my decision to use old version,
unfortunately), I'm getting an error:

Spetsnaz_AI.obj : error LNK2001: unresolved external symbol "struct _object
* __cdecl boost::python::detail::init_module(char const *,void
(__cdecl*)(void))" (?init_module at detail@python at boost@@YAPAU_object@
@PBDP6AXXZ at Z)

I've got it down to where this is the only unresolved external.  My export
code is viewable:

http://pastebin.com/m439861b0 .


I'm not sure what the error implies.  Is the init_module really an external
symbol that gets linked?  Or does that get defined in my code?  I tried
examining the definition of the BOOST_PYTHON_MODULE macro and it seemed to
indicate the latter.  Would appreciate any help or redirection.

Many thanks,
Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100216/a40c76bd/attachment.htm>

From amundson at fnal.gov  Thu Feb 18 23:24:57 2010
From: amundson at fnal.gov (James Amundson)
Date: Thu, 18 Feb 2010 16:24:57 -0600
Subject: [C++-sig] Boost python exception translation failure on BlueGene/P
Message-ID: <4B7DBE39.6090703@fnal.gov>

Boost Python seems to be failing to catch and translate C++ exceptions 
for me on a BlueGene/P installation. I really don't know what to do next 
in debugging the problem, so I would appreciate any advice.

I have a simple test module, pyexcept
------------------------------------------------------------
#include <boost/python.hpp>
#include <iostream>
#include <stdexcept>


void
foo()
{
     std::cout << "in foo, about to raise exception\n";
     throw std::runtime_error("runtime_error from foo");
     std::cout << "this should never be seen\n";
}

using namespace boost::python;

BOOST_PYTHON_MODULE(pyexcept)
{
     def("foo",foo);
}
------------------------------------------------------------

I test it with the following python script:
------------------------------------------------------------
#!/usr/bin/env python

import pyexcept

print "about to run pyexcept.foo, catching exception"
try:
     pyexcept.foo()
except RuntimeError,e:
     print "caught RuntimeError,",e
------------------------------------------------------------

On my Linux machine I see:
------------------------------------------------------------
about to run pyexcept.foo, catching exception
in foo, about to raise exception
caught RuntimeError, runtime_error from foo
------------------------------------------------------------

On the BlueGene/P machine I see:
------------------------------------------------------------
about to run pyexcept.foo, catching exception
in foo, about to raise exception
terminate called after throwing an instance of 'std::runtime_error'
   what():  runtime_error from foo
------------------------------------------------------------

I have (cross-)compiled boost myself using the system's installed 
version of the gnu compilers, 4.1.2:
------------------------------------------------------------
|login2>mpicxx.gnu --version
powerpc-bgp-linux-g++ (GCC) 4.1.2 (BGP)
------------------------------------------------------------

(The mpicxx.gnu script is a wrapper around the cross-compiling g++.) Has 
anyone seen a problem like this? Any ideas as to how to debug it?

Thanks for any advice.

--Jim Amundson


From rwgk at yahoo.com  Fri Feb 19 00:19:17 2010
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Thu, 18 Feb 2010 15:19:17 -0800 (PST)
Subject: [C++-sig] Boost python exception translation failure on
	BlueGene/P
In-Reply-To: <4B7DBE39.6090703@fnal.gov>
References: <4B7DBE39.6090703@fnal.gov>
Message-ID: <269988.77291.qm@web111415.mail.gq1.yahoo.com>

Your problem may be that an exception thrown in one .so isn't correctly caught in another .so.

In the dark past we had problems like this, too. To this day I don't import extensions
directly, but use a small wrapper function, import_ext(). The essential bit is

    sys.setdlopenflags(0x100|0x2)

The hex values are platform specific. IIRC it is RTLD_NOW | RTLD_GLOBAL (may be backwards).
You want to insert something like this right before importing your extension.


Full code:

http://cci.lbl.gov/cctbx_sources/boost_adaptbx/boost/python.py

Example boilerplate code for imports:

import boost.python
ext = boost.python.import_ext("scitbx_lbfgs_ext")
from scitbx_lbfgs_ext import *

From amundson at fnal.gov  Fri Feb 19 05:28:17 2010
From: amundson at fnal.gov (James Amundson)
Date: Thu, 18 Feb 2010 22:28:17 -0600
Subject: [C++-sig] Boost python exception translation failure
	on	BlueGene/P
In-Reply-To: <269988.77291.qm@web111415.mail.gq1.yahoo.com>
References: <4B7DBE39.6090703@fnal.gov>
	<269988.77291.qm@web111415.mail.gq1.yahoo.com>
Message-ID: <4B7E1361.7070108@fnal.gov>

On 02/18/2010 05:19 PM, Ralf W. Grosse-Kunstleve wrote:
> Your problem may be that an exception thrown in one .so isn't correctly caught in another .so.
>    
Hmmm. I only have one module in my test -- pyexcept. Is the fact that it 
has to link to libboost_python enough to cause that problem?
> In the dark past we had problems like this, too. To this day I don't import extensions
> directly, but use a small wrapper function, import_ext(). The essential bit is
>
>      sys.setdlopenflags(0x100|0x2)
>
> The hex values are platform specific. IIRC it is RTLD_NOW | RTLD_GLOBAL (may be backwards).
> You want to insert something like this right before importing your extension.
>    
OK. I tried this:
---------------------------------------------------------------------------------
#!/usr/bin/env python
import sys
import dl
sys.setdlopenflags(dl.RTLD_NOW | dl.RTLD_GLOBAL)
import pyexcept

print "about to run pyexcept.foo, catching exception"
try:
     pyexcept.foo()
except RuntimeError,e:
     print "caught RuntimeError,",e

---------------------------------------------------------------------------------

I saw the same behaviour as before -- I never catch the exception.

Thanks for the suggestion, though. It was new to me.

--Jim Amundson

From rwgk at yahoo.com  Fri Feb 19 09:43:20 2010
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Fri, 19 Feb 2010 00:43:20 -0800 (PST)
Subject: [C++-sig] Boost python exception translation failure on
	BlueGene/P
In-Reply-To: <4B7E1361.7070108@fnal.gov>
References: <4B7DBE39.6090703@fnal.gov>
	<269988.77291.qm@web111415.mail.gq1.yahoo.com>
	<4B7E1361.7070108@fnal.gov>
Message-ID: <913881.22737.qm@web111403.mail.gq1.yahoo.com>

> Hmmm. I only have one module in my test -- pyexcept. Is the fact that it has to link to
> libboost_python enough to cause that problem?


Yes. (It was the motivation for boost::python::throw_error_already_set() ).
Sorry to hear my suggestion didn't help. I hope you'll get help from people
familiar with IBM's environment.
Ralf

From nitroamos at gmail.com  Sat Feb 20 00:09:32 2010
From: nitroamos at gmail.com (Amos Anderson)
Date: Fri, 19 Feb 2010 15:09:32 -0800
Subject: [C++-sig] how do i interrupt a C++ extension?
Message-ID: <9e910971002191509w1973a597nacdcf6c6f1b99cc0@mail.gmail.com>

>>> ----- Original Message ----
>>> From: Amos Anderson <nitroamos at gmail.com>
>>> To: Development of Python/C++ integration <cplusplus-sig at python.org>
>>> Sent: Wed, February 3, 2010 12:24:13 PM
>>> Subject: [C++-sig] how do i interrupt a C++ extension?
>>>
>>> Hello --
>>>
>>> I've got a python script with C++ extensions. Some of my extensions
>>> take a long time to complete, and I don't want to wait for them to
>>> finish when I'm debugging stuff. However, when I do Ctrl-C in my
>>> terminal, it's completely ignored. So it looks like python is trapping
>>> the signal, but apparently can't do anything with it until the
>>> extension returns control to the python script. I guess ideally,
>>> Ctrl-C would kill the extension and return control to python,
>>> generating an exception, but I'd also be ok if Ctrl-C killed the
>>> python script too.
>>>
>>> I've been googling around, but can't figure out how this seemingly
>>> simple (and desired) task is accomplished. Anybody know how to do it?
>>> Right now, the only solution is Ctrl-Z and kill %1 so I guess that
>>> works for now...
>>>
>>> thanks!
>>>
>>> Amos.


Thanks for all the suggestions, but I just figured out how to do what
I want, and I figured I'd share it with everybody. This was my
coworker's idea. I just wrap my extension in a separate Process, which
can then be killed by the original process as needed. For example:


from multiprocessing import Process
...
      p = Process(target=minimizeStructure, args=(filename,))
      p.start()
      try:
         p.join()
      except:
         p.terminate()
         print "Error: early termination."


and this does exactly what I need.

Amos.

From a.h.prins at gmail.com  Mon Feb 22 15:45:07 2010
From: a.h.prins at gmail.com (=?ISO-8859-1?Q?Andr=E9_Prins?=)
Date: Mon, 22 Feb 2010 15:45:07 +0100
Subject: [C++-sig] Duplicate class and method definition
Message-ID: <2fc5b3ca1002220645u2175fac8ud07b8e22ce738daa@mail.gmail.com>

Hi All,

Recently, I had some problems with a duplicate method and class
definition begin accepted by boost::python (version 1.41 on MSVC).
Consider the following c++-code, which declares a class Foo twice, but
with different bindings for the same function.

/* C++ code: */
#include <iostream>
#include <boost/python.hpp>
using namespace boost::python;

struct Foo
{
    void f( int m ){ std::cerr << "F set to " << m  << std::endl;  }
    void g( int m ){ std::cerr << "G set to " << 10*m << std::endl; }
};

BOOST_PYTHON_MODULE( FooPython )
{
    // First definition binding 'g'
    class_< Foo, boost::noncopyable >( "Foo" )
        .def( "func", &Foo::f )
        ;
    // Second definition binding 'f'
    class_< Foo, boost::noncopyable >( "Foo" )
        .def( "func", &Foo::g )
        ;
}

# Python-code
import FooPython
f = FooPython.Foo()
f.func( 3 )

This outputs the line "G set to 30" and the original function-binding
(to f) is gone. When I remove the boost::noncopyable when creating the
python-binding Foo, I get an assert upon importing FooPython:
    file: libs\python\src\converter\registry.cpp @ line 212
    expression: slot->m_to_python == 0

It is obvious that this is not the intended use of Boost-python, but I
am wondering: why does it assert at runtime without the noncopyable
and why does it "more or less work" with the boost::noncopyable. And
is there a method to detect such multiple definitions of the same
class.

Kind Regards,
Andr?

From seefeld at sympatico.ca  Mon Feb 22 15:59:08 2010
From: seefeld at sympatico.ca (Stefan Seefeld)
Date: Mon, 22 Feb 2010 09:59:08 -0500
Subject: [C++-sig] Duplicate class and method definition
In-Reply-To: <2fc5b3ca1002220645u2175fac8ud07b8e22ce738daa@mail.gmail.com>
References: <2fc5b3ca1002220645u2175fac8ud07b8e22ce738daa@mail.gmail.com>
Message-ID: <4B829BBC.4000300@sympatico.ca>

On 02/22/2010 09:45 AM, Andr? Prins wrote:
> Hi All,
>
> Recently, I had some problems with a duplicate method and class
> definition begin accepted by boost::python (version 1.41 on MSVC).
> Consider the following c++-code, which declares a class Foo twice, but
> with different bindings for the same function.
>    

[...]

> It is obvious that this is not the intended use of Boost-python, but I
> am wondering: why does it assert at runtime without the noncopyable
> and why does it "more or less work" with the boost::noncopyable. And
> is there a method to detect such multiple definitions of the same
> class.
>    

For the first two questions: I would suggest you consider the code 
"ill-formed" and the behavior "undefined", so there isn't any need to 
(second-)guess what the library is doing, and why things appear to be 
working in one case but not the other.

As to the last question: I suggest you submit a request for enhancement 
for this. I'm pretty sure the library can detect this, and raise an 
appropriate exception. Alternatively, if it is decided that the above is 
a valuable use-case to support, the library needs to be improved to 
support it.

I'm saying that because recently we had a discussion about per-module 
converter registries, to avoid conflicts when working with multiple 
extension modules that may provide equivalent converters, where it is 
important to be able to pick explicitly which one is going to be used.

Thanks,
         Stefan


-- 

       ...ich hab' noch einen Koffer in Berlin...


From amundson at fnal.gov  Wed Feb 24 18:37:22 2010
From: amundson at fnal.gov (James Amundson)
Date: Wed, 24 Feb 2010 11:37:22 -0600
Subject: [C++-sig] Boost python exception translation failure on
	BlueGene/P
In-Reply-To: <4B7DBE39.6090703@fnal.gov>
References: <4B7DBE39.6090703@fnal.gov>
Message-ID: <4B8563D2.3030805@fnal.gov>


On 02/18/2010 04:24 PM, James Amundson wrote:
> Boost Python seems to be failing to catch and translate C++ exceptions 
> for me on a BlueGene/P installation. I really don't know what to do 
> next in debugging the problem, so I would appreciate any advice.

I have finally solved this problem myself -- I am responding to my own 
message on these mailing lists in case this information is useful to 
others.

In the end the solution was to add the flag "-dynamic" to the 
libboost_python link line. Without the flag, boost::python mostly works 
on BlueGene/P, but exceptions are not caught. With the flag, 
boost_python passes all tests I have done.

I spent some time trying to understand how to tell bjam to add a flag to 
the boost_python link line, but I gave up after about fifteen minutes. 
With the CMake build of boost, it was easy: I simply did
     cmake -DCMAKE_SHARED_LINKER_FLAGS="-dynamic"
Overall, I found dealing with the cmake build of boost so much more 
pleasant than dealing with bjam that I don't understand why the CMake 
build hasn't taken over the mainstream yet.

--Jim Amundson

P.S. The original problem report:
> I have a simple test module, pyexcept
> ------------------------------------------------------------
> #include <boost/python.hpp>
> #include <iostream>
> #include <stdexcept>
>
>
> void
> foo()
> {
>     std::cout << "in foo, about to raise exception\n";
>     throw std::runtime_error("runtime_error from foo");
>     std::cout << "this should never be seen\n";
> }
>
> using namespace boost::python;
>
> BOOST_PYTHON_MODULE(pyexcept)
> {
>     def("foo",foo);
> }
> ------------------------------------------------------------
>
> I test it with the following python script:
> ------------------------------------------------------------
> #!/usr/bin/env python
>
> import pyexcept
>
> print "about to run pyexcept.foo, catching exception"
> try:
>     pyexcept.foo()
> except RuntimeError,e:
>     print "caught RuntimeError,",e
> ------------------------------------------------------------
>
> On my Linux machine I see:
> ------------------------------------------------------------
> about to run pyexcept.foo, catching exception
> in foo, about to raise exception
> caught RuntimeError, runtime_error from foo
> ------------------------------------------------------------
>
> On the BlueGene/P machine I see:
> ------------------------------------------------------------
> about to run pyexcept.foo, catching exception
> in foo, about to raise exception
> terminate called after throwing an instance of 'std::runtime_error'
>   what():  runtime_error from foo
> ------------------------------------------------------------
>
> I have (cross-)compiled boost myself using the system's installed 
> version of the gnu compilers, 4.1.2:
> ------------------------------------------------------------
> |login2>mpicxx.gnu --version
> powerpc-bgp-linux-g++ (GCC) 4.1.2 (BGP)
> ------------------------------------------------------------
>
> (The mpicxx.gnu script is a wrapper around the cross-compiling g++.) 
> Has anyone seen a problem like this? Any ideas as to how to debug it?
>
> Thanks for any advice.
>
> --Jim Amundson
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig


From arhipjan at gmail.com  Thu Feb 25 12:40:10 2010
From: arhipjan at gmail.com (Alexander Arhipenko)
Date: Thu, 25 Feb 2010 13:40:10 +0200
Subject: [C++-sig] Boost python exception translation failure on
	BlueGene/P
In-Reply-To: <4B8563D2.3030805@fnal.gov>
References: <4B7DBE39.6090703@fnal.gov> <4B8563D2.3030805@fnal.gov>
Message-ID: <16404e4a1002250340t5acb71e5g434953992573cdd9@mail.gmail.com>

On Wed, Feb 24, 2010 at 7:37 PM, James Amundson <amundson at fnal.gov> wrote:
>
> On 02/18/2010 04:24 PM, James Amundson wrote:
>>
[snip]
> I spent some time trying to understand how to tell bjam to add a flag to the
> boost_python link line, but I gave up after about fifteen minutes. With the
> CMake build of boost, it was easy: I simply did
> ? ?cmake -DCMAKE_SHARED_LINKER_FLAGS="-dynamic"
> Overall, I found dealing with the cmake build of boost so much more pleasant
> than dealing with bjam that I don't understand why the CMake build hasn't
> taken over the mainstream yet.
>
> --Jim Amundson
>

Hi Jim,
You could simply type in you Jamfile:

<link>shared:<linkflags>="-dynamic"

Regards

From nitroamos at gmail.com  Sat Feb 27 00:47:22 2010
From: nitroamos at gmail.com (Amos Anderson)
Date: Fri, 26 Feb 2010 15:47:22 -0800
Subject: [C++-sig] incorrect linking to libpython
Message-ID: <9e910971002261547j7696c51fk83401ae40996f2d7@mail.gmail.com>

Hello --

summary: I'm trying to compile the boost python library using a custom
installed python on osx. however, it appears to be linking to the
system library, not the installed library.


I'm using OSX 10.6.2, which has python 2.6.1 by default. I wanted to
try to install 2.6.4 just to see how the process works. I
To install python, I did this:
./configure  --enable-framework
make
make install

which put the distribution here:
/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6

I changed a boost config file to read:
using python
      : 2.6
      : /Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
      : /Library/Frameworks/Python.framework/Versions/2.6/include/python2.6
      : /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config ;

and compiled the library. However, when I examine the library itself:
% otool -L libboost_python.dylib
libboost_python.dylib:
	libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
	/System/Library/Frameworks/Python.framework/Versions/2.6/Python
(compatibility version 2.6.0, current version 2.6.1)
	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 125.0.0)

I can see that it linked to the system's version. Why did it choose
that one? The problem is that when I run my code, I get the error:
Fatal Python error: Interpreter not initialized (version mismatch?)
../../../triad.sh: line 9: 59136 Abort trap              python $@


the OS gives me a popup window mentioning all the linked libraries:
Binary Images:
       0x100000000 -        0x100000ff7 +org.python.python 2.6.4
(2.6.4) <536A2002-9AEC-D7DF-8C2B-6C634FB7F37E>
/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python
       0x100003000 -        0x100153fe7 +org.python.python 2.6.4, (c)
2004-2008 Python Software Foundation. (2.6.4)
<3B4D8CE1-8D9C-97CB-57BA-A8F4C40CE5F0>
/Library/Frameworks/Python.framework/Versions/2.6/Python
       0x1002e7000 -        0x1002e8ff7 +time.so ??? (???)
<CB23D26C-1417-4D31-9E06-1E33CCDCA83C>
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/time.so
       0x100480000 -        0x100493fef +libgcc_s.1.dylib ??? (???)
<E51F2952-8BEB-C427-10A9-672FC10E3A54> /usr/local/lib/libgcc_s.1.dylib
       0x100753000 -        0x10077bfff +libboost_wserialization.dylib
??? (???) <4FD94C16-75D0-DE5D-A27B-E3AEAEEA8CCC>
/Users/amos/triad/trunk/lib/libboost_wserialization.dylib
       0x101000000 -        0x1013eeff7 +triad.so ??? (???)
<EEF65664-60F0-E721-06A8-82F88CEA1EFC>
/Users/amos/triad/trunk/lib/triad.so
       0x1020eb000 -        0x102124fef +libboost_serialization.dylib
??? (???) <33DB6DD1-4712-AEE1-0761-18054566A5E4>
/Users/amos/triad/trunk/lib/libboost_serialization.dylib
       0x1021b1000 -        0x1021f0ff7 +libboost_python.dylib ???
(???) <67258016-C11E-8EDF-BF36-103DBC219C78>
/Users/amos/triad/trunk/lib/libboost_python.dylib
       0x10223e000 -        0x10234eff7  org.python.python 2.6.1
(2.6.1) <01EBD0D4-F181-045F-C7A8-B7B3C5D4204F>
/System/Library/Frameworks/Python.framework/Versions/2.6/Python
    0x7fff5fc00000 -     0x7fff5fc3bdef  dyld 132.1 (???)
<B633F790-4DDB-53CD-7ACF-2A3682BCEA9F> /usr/lib/dyld
    0x7fff813dc000 -     0x7fff81459fef  libstdc++.6.dylib ??? (???)
<35ECA411-2C08-FD7D-11B1-1B7A04921A5C> /usr/lib/libstdc++.6.dylib
    0x7fff81f2c000 -     0x7fff81f3dfef  libz.1.dylib ??? (???)
<3A7A4C48-A4C8-A78A-8B87-C0DDF6601AC8> /usr/lib/libz.1.dylib
    0x7fff846e0000 -     0x7fff846e4ff7  libmathCommon.A.dylib ???
(???) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5>
/usr/lib/system/libmathCommon.A.dylib
    0x7fff888bb000 -     0x7fff88a79ff7  libSystem.B.dylib ??? (???)
<526DD3E5-2A8B-4512-ED97-01B832369959> /usr/lib/libSystem.B.dylib
    0x7fffffe00000 -     0x7fffffe01fff  libSystem.B.dylib ??? (???)
<526DD3E5-2A8B-4512-ED97-01B832369959> /usr/lib/libSystem.B.dylib

Where i've deleted all the lines with my libraries. Note: I can do this:
/Library/Frameworks> sudo mv Python.Framework temp.Python.Framework
and then everything works fine. So I assume that the problem is the
one python file 2.6.1 which didn't come from the same python path as
the other python paths. When I compile, I get this:
% bjam -d 2 release
...
deleted similar lines
....
    "g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline
-Wall -dynamic -no-cpp-precomp -gdwarf-2 -fPIC  -DBOOST_ALL_NO_LIB=1
-DBOOST_PYTHON_SOURCE -DNDEBUG
-I"/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6"
-I"boost/boost_1_40_0" -c -o
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/import.o"
"boost/boost_1_40_0/libs/python/src/import.cpp"

darwin.compile.c++
boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/exec.o

    "g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline
-Wall -dynamic -no-cpp-precomp -gdwarf-2 -fPIC  -DBOOST_ALL_NO_LIB=1
-DBOOST_PYTHON_SOURCE -DNDEBUG
-I"/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6"
-I"boost/boost_1_40_0" -c -o
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/exec.o"
"boost/boost_1_40_0/libs/python/src/exec.cpp"

darwin.compile.c++
boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/function_doc_signature.o

    "g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline
-Wall -dynamic -no-cpp-precomp -gdwarf-2 -fPIC  -DBOOST_ALL_NO_LIB=1
-DBOOST_PYTHON_SOURCE -DNDEBUG
-I"/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6"
-I"boost/boost_1_40_0" -c -o
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/function_doc_signature.o"
"boost/boost_1_40_0/libs/python/src/object/function_doc_signature.cpp"

darwin.link.dll
boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/libboost_python.dylib

    "g++" -dynamiclib -Wl,-single_module -install_name
"libboost_python.dylib"
-L"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config"
-o "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/libboost_python.dylib"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/numeric.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/list.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/long.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/dict.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/tuple.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/str.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/slice.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/converter/from_python.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/converter/registry.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/converter/type_id.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/enum.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/class.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/function.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/inheritance.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/life_support.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/pickle_support.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/errors.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/module.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/converter/builtin_converters.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/converter/arg_to_python_base.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/iterator.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/stl_iterator.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object_protocol.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object_operators.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/wrapper.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/import.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/exec.o"
"boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/object/function_doc_signature.o"
 -lpython2.6    -headerpad_max_install_names -Wl,-dead_strip
-no_dead_strip_inits_and_terms

So again we can see that it's finding the header files for each object
file, but then when they're all being linked, it's seems to be
skipping the library that (I think) it should be picking:
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config> ls -lrt
total 208
-rw-r--r--  1 root  admin   3.7K Feb 26 12:25 python.o
-rwxr-xr-x@ 1 root  admin   7.3K Feb 26 12:25 makesetup*
-rwxr-xr-x@ 1 root  admin   7.0K Feb 26 12:25 install-sh*
-rw-r--r--@ 1 root  admin   1.4K Feb 26 12:25 config.c.in
-rw-r--r--  1 root  admin   2.0K Feb 26 12:25 config.c
-rw-r--r--  1 root  admin    41B Feb 26 12:25 Setup.local
-rw-r--r--  1 root  admin   368B Feb 26 12:25 Setup.config
-rw-r--r--@ 1 root  admin    18K Feb 26 12:25 Setup
-rw-r--r--  1 root  admin    42K Feb 26 12:25 Makefile
lrwxr-xr-x  1 root  admin    15B Feb 26 12:25 libpython2.6.a@ -> ../../../Python


I tried changing the name of the library to something else more unique
and manually ran the link line (outside of bjam), and I *can* link to
it, so that's not the problem. Why isn't it choosing the -L selected
libpython2.6.a? Or maybe I'm doing something else wrong. Or maybe it's
Apple's fault, or maybe I need to update boost from 1.40.


amos.

From blake at hailmail.net  Sat Feb 27 03:36:16 2010
From: blake at hailmail.net (Dane Springmeyer)
Date: Fri, 26 Feb 2010 18:36:16 -0800
Subject: [C++-sig] incorrect linking to libpython
In-Reply-To: <9e910971002261547j7696c51fk83401ae40996f2d7@mail.gmail.com>
References: <9e910971002261547j7696c51fk83401ae40996f2d7@mail.gmail.com>
Message-ID: <57569986-7D90-4686-8384-809F8D27FC68@hailmail.net>

Amos,

I feel your pain, seriously. I've been down many of these paths as well.

My overall sense is that direct linking of boost to a python  
executable may not be necessary, and rather that bjam could use `- 
undefined dynamic_lookup`, which will mean that when you import the  
python module you've built with boost::python the right version of  
python will be dynamically looked up based on the interpreter you are  
running. There are certainly potential problems with this route (and  
I've only lightly tested so far) but I think it may hold promise.

More to the point, I've also got into the habit of passing -Z to the  
linker to make sure that if the custom python framework is not able to  
be linked, ld does not fallback to linking to the "system" or apple  
provided python framework and rather throws an error so you know the  
exact point things went wrong rather than only later by getting the  
dreaded "Version Mismatch" error.

Failing to link to the /Library or user installed python26 is almost  
invariably the case on snow leopard because two copies will exist and  
the user installed version won't be preferentially linked with the  
normal -L/path/to/dir -lpython2.6 method because 'libpython2.6' is  
actually:

/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config/ 
libpython2.6.a

and my sense is the linker will prefer the /usr/lib/libpython2.6.dylib  
over that because of the 'a' (even though it is actually, and oddly, a  
symlink to the dynamic library at:

/Library/Frameworks/Python.framework/Versions/2.6/Python

So, one workaround is symlinking:

/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/config/ 
libpython2.6.dylib  -> ../../../Python

Then bjam should be able to find it.

But I think a preferable method would be if bjam actually sent the  
linker line:

-F/Library/Library/Frameworks -framework Python -Z

instead of -L/etc/etc -lpython...


Dane


More info on those ld commands at:
http://developer.apple.com/mac/library/DOCUMENTATION/Darwin/Reference/ManPages/man1/ld.1.html



On Feb 26, 2010, at 3:47 PM, Amos Anderson wrote:

> Hello --
>
> summary: I'm trying to compile the boost python library using a custom
> installed python on osx. however, it appears to be linking to the
> system library, not the installed library.
>
>
> I'm using OSX 10.6.2, which has python 2.6.1 by default. I wanted to
> try to install 2.6.4 just to see how the process works. I
> To install python, I did this:
> ./configure  --enable-framework
> make
> make install
>
> which put the distribution here:
> /Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
>
> I changed a boost config file to read:
> using python
>      : 2.6
>      : /Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
>      : /Library/Frameworks/Python.framework/Versions/2.6/include/ 
> python2.6
>      : /Library/Frameworks/Python.framework/Versions/2.6/lib/ 
> python2.6/config ;
>
> and compiled the library. However, when I examine the library itself:
> % otool -L libboost_python.dylib
> libboost_python.dylib:
> 	libboost_python.dylib (compatibility version 0.0.0, current version  
> 0.0.0)
> 	/System/Library/Frameworks/Python.framework/Versions/2.6/Python
> (compatibility version 2.6.0, current version 2.6.1)
> 	/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current  
> version 7.9.0)
> 	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
> version 125.0.0)
>
> I can see that it linked to the system's version. Why did it choose
> that one? The problem is that when I run my code, I get the error:
> Fatal Python error: Interpreter not initialized (version mismatch?)
> ../../../triad.sh: line 9: 59136 Abort trap              python $@
>
>
> the OS gives me a popup window mentioning all the linked libraries:
> Binary Images:
>       0x100000000 -        0x100000ff7 +org.python.python 2.6.4
> (2.6.4) <536A2002-9AEC-D7DF-8C2B-6C634FB7F37E>
> /Library/Frameworks/Python.framework/Versions/2.6/Resources/ 
> Python.app/Contents/MacOS/Python
>       0x100003000 -        0x100153fe7 +org.python.python 2.6.4, (c)
> 2004-2008 Python Software Foundation. (2.6.4)
> <3B4D8CE1-8D9C-97CB-57BA-A8F4C40CE5F0>
> /Library/Frameworks/Python.framework/Versions/2.6/Python
>       0x1002e7000 -        0x1002e8ff7 +time.so ??? (???)
> <CB23D26C-1417-4D31-9E06-1E33CCDCA83C>
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib- 
> dynload/time.so
>       0x100480000 -        0x100493fef +libgcc_s.1.dylib ??? (???)
> <E51F2952-8BEB-C427-10A9-672FC10E3A54> /usr/local/lib/libgcc_s.1.dylib
>       0x100753000 -        0x10077bfff +libboost_wserialization.dylib
> ??? (???) <4FD94C16-75D0-DE5D-A27B-E3AEAEEA8CCC>
> /Users/amos/triad/trunk/lib/libboost_wserialization.dylib
>       0x101000000 -        0x1013eeff7 +triad.so ??? (???)
> <EEF65664-60F0-E721-06A8-82F88CEA1EFC>
> /Users/amos/triad/trunk/lib/triad.so
>       0x1020eb000 -        0x102124fef +libboost_serialization.dylib
> ??? (???) <33DB6DD1-4712-AEE1-0761-18054566A5E4>
> /Users/amos/triad/trunk/lib/libboost_serialization.dylib
>       0x1021b1000 -        0x1021f0ff7 +libboost_python.dylib ???
> (???) <67258016-C11E-8EDF-BF36-103DBC219C78>
> /Users/amos/triad/trunk/lib/libboost_python.dylib
>       0x10223e000 -        0x10234eff7  org.python.python 2.6.1
> (2.6.1) <01EBD0D4-F181-045F-C7A8-B7B3C5D4204F>
> /System/Library/Frameworks/Python.framework/Versions/2.6/Python
>    0x7fff5fc00000 -     0x7fff5fc3bdef  dyld 132.1 (???)
> <B633F790-4DDB-53CD-7ACF-2A3682BCEA9F> /usr/lib/dyld
>    0x7fff813dc000 -     0x7fff81459fef  libstdc++.6.dylib ??? (???)
> <35ECA411-2C08-FD7D-11B1-1B7A04921A5C> /usr/lib/libstdc++.6.dylib
>    0x7fff81f2c000 -     0x7fff81f3dfef  libz.1.dylib ??? (???)
> <3A7A4C48-A4C8-A78A-8B87-C0DDF6601AC8> /usr/lib/libz.1.dylib
>    0x7fff846e0000 -     0x7fff846e4ff7  libmathCommon.A.dylib ???
> (???) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5>
> /usr/lib/system/libmathCommon.A.dylib
>    0x7fff888bb000 -     0x7fff88a79ff7  libSystem.B.dylib ??? (???)
> <526DD3E5-2A8B-4512-ED97-01B832369959> /usr/lib/libSystem.B.dylib
>    0x7fffffe00000 -     0x7fffffe01fff  libSystem.B.dylib ??? (???)
> <526DD3E5-2A8B-4512-ED97-01B832369959> /usr/lib/libSystem.B.dylib
>
> Where i've deleted all the lines with my libraries. Note: I can do  
> this:
> /Library/Frameworks> sudo mv Python.Framework temp.Python.Framework
> and then everything works fine. So I assume that the problem is the
> one python file 2.6.1 which didn't come from the same python path as
> the other python paths. When I compile, I get this:
> % bjam -d 2 release
> ...
> deleted similar lines
> ....
>    "g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline
> -Wall -dynamic -no-cpp-precomp -gdwarf-2 -fPIC  -DBOOST_ALL_NO_LIB=1
> -DBOOST_PYTHON_SOURCE -DNDEBUG
> -I"/Library/Frameworks/Python.framework/Versions/2.6/include/ 
> python2.6"
> -I"boost/boost_1_40_0" -c -o
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> import.o"
> "boost/boost_1_40_0/libs/python/src/import.cpp"
>
> darwin.compile.c++
> boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> exec.o
>
>    "g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline
> -Wall -dynamic -no-cpp-precomp -gdwarf-2 -fPIC  -DBOOST_ALL_NO_LIB=1
> -DBOOST_PYTHON_SOURCE -DNDEBUG
> -I"/Library/Frameworks/Python.framework/Versions/2.6/include/ 
> python2.6"
> -I"boost/boost_1_40_0" -c -o
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> exec.o"
> "boost/boost_1_40_0/libs/python/src/exec.cpp"
>
> darwin.compile.c++
> boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/function_doc_signature.o
>
>    "g++"  -ftemplate-depth-128 -O3 -finline-functions -Wno-inline
> -Wall -dynamic -no-cpp-precomp -gdwarf-2 -fPIC  -DBOOST_ALL_NO_LIB=1
> -DBOOST_PYTHON_SOURCE -DNDEBUG
> -I"/Library/Frameworks/Python.framework/Versions/2.6/include/ 
> python2.6"
> -I"boost/boost_1_40_0" -c -o
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/function_doc_signature.o"
> "boost/boost_1_40_0/libs/python/src/object/function_doc_signature.cpp"
>
> darwin.link.dll
> boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> libboost_python.dylib
>
>    "g++" -dynamiclib -Wl,-single_module -install_name
> "libboost_python.dylib"
> -L"/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ 
> config"
> -o "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> libboost_python.dylib"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> numeric.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> list.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> long.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> dict.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> tuple.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> str.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> slice.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> converter/from_python.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> converter/registry.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> converter/type_id.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/enum.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/class.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/function.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/inheritance.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/life_support.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/pickle_support.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> errors.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> module.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> converter/builtin_converters.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> converter/arg_to_python_base.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/iterator.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/stl_iterator.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object_protocol.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object_operators.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> wrapper.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> import.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> exec.o"
> "boost/boost_1_40_0/bin.v2/libs/python/build/darwin-4.2.1/release/ 
> object/function_doc_signature.o"
> -lpython2.6    -headerpad_max_install_names -Wl,-dead_strip
> -no_dead_strip_inits_and_terms
>
> So again we can see that it's finding the header files for each object
> file, but then when they're all being linked, it's seems to be
> skipping the library that (I think) it should be picking:
> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ 
> config> ls -lrt
> total 208
> -rw-r--r--  1 root  admin   3.7K Feb 26 12:25 python.o
> -rwxr-xr-x@ 1 root  admin   7.3K Feb 26 12:25 makesetup*
> -rwxr-xr-x@ 1 root  admin   7.0K Feb 26 12:25 install-sh*
> -rw-r--r--@ 1 root  admin   1.4K Feb 26 12:25 config.c.in
> -rw-r--r--  1 root  admin   2.0K Feb 26 12:25 config.c
> -rw-r--r--  1 root  admin    41B Feb 26 12:25 Setup.local
> -rw-r--r--  1 root  admin   368B Feb 26 12:25 Setup.config
> -rw-r--r--@ 1 root  admin    18K Feb 26 12:25 Setup
> -rw-r--r--  1 root  admin    42K Feb 26 12:25 Makefile
> lrwxr-xr-x  1 root  admin    15B Feb 26 12:25 libpython2.6.a@ - 
> > ../../../Python
>
>
> I tried changing the name of the library to something else more unique
> and manually ran the link line (outside of bjam), and I *can* link to
> it, so that's not the problem. Why isn't it choosing the -L selected
> libpython2.6.a? Or maybe I'm doing something else wrong. Or maybe it's
> Apple's fault, or maybe I need to update boost from 1.40.
>
>
> amos.
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig


From alexey.akimov85 at gmail.com  Sun Feb 28 04:52:17 2010
From: alexey.akimov85 at gmail.com (Alexey Akimov)
Date: Sat, 27 Feb 2010 21:52:17 -0600
Subject: [C++-sig] profiling python extension
Message-ID: <382d227f1002271952p25858f1eyeb1c495bd282f2b@mail.gmail.com>

Dear all,

Does anyone know how one may profile a python extention? There is a bunch of
literature on profiling of pure binaries (programs and even libraries), pure
python code, etc. However the profiling of a python module from python
script seems a tricky procedure.
What I tried is to pass variant=profile to bjam. Everything works fine and I
get .so library corresponding to my module (placed into /profile
sub-directory of this particular build). When I execute the python script
invoking my module's functions (performance of those functions is most
interesting for me) there is no any output generated (something similar to
gmon.out). So that is basically my question - how can I retrieve the
profiling information about my extension? Of course I can compile a normal
executable using the functions in the module and then use a standard
approach (and of course gprof), but I hope that there should be more
straightforward and easy way to do this (hopefully by passing some arguments
to bjam).
Thanks in advance.

Alexey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100227/dc1aecd6/attachment.html>

From rwgk at yahoo.com  Sun Feb 28 06:44:15 2010
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Sat, 27 Feb 2010 21:44:15 -0800 (PST)
Subject: [C++-sig] profiling python extension
In-Reply-To: <382d227f1002271952p25858f1eyeb1c495bd282f2b@mail.gmail.com>
References: <382d227f1002271952p25858f1eyeb1c495bd282f2b@mail.gmail.com>
Message-ID: <47466.76886.qm@web111404.mail.gq1.yahoo.com>

I usually use oprofile. Typically, I write a small piece of python to call the
extension multiple times so it runs the critical code for about a minute (10
seconds will probably do, too), then the extension code is the only thing
showing up near the top of the oprofile stats.
It is important that nothing else is running on the machine when you run
your extension for profiling, since oprofile is a kernel-level tool and sees
the entire system.
Last time I looked at the docs (ca. 5 years ago...) you had to be root for
"opcontrol --start". I'm not sure if that's still a requirement.
The commands I use are:

opcontrol --reset; opcontrol --start # root
# run as usual (not root)
opcontrol --shutdown # root
opreport -l > report
opannotate --source > annotated
grep '^[ 0-9][ 0-9][ 0-9][ 0-9][ 0-9][0-9] ' annotated | sort -n

You may need "yum install oprofile" or similar.



________________________________
From: Alexey Akimov <alexey.akimov85 at gmail.com>
To: Development of Python/C++ integration <cplusplus-sig at python.org>
Sent: Sat, February 27, 2010 7:52:17 PM
Subject: [C++-sig] profiling python extension

Dear all,

Does anyone know how one may profile a python extention? There is a bunch of literature on profiling of pure binaries (programs and even libraries), pure python code, etc. However the profiling of a python module from python script seems a tricky procedure. 
What I tried is to pass variant=profile to bjam. Everything works fine and I get .so library corresponding to my module (placed into /profile sub-directory of this particular build). When I execute the python script invoking my module's functions (performance of those functions is most interesting for me) there is no any output generated (something similar to gmon.out). So that is basically my question - how can I retrieve the profiling information about my extension? Of course I can compile a normal executable using the functions in the module and then use a standard approach (and of course gprof), but I hope that there should be more straightforward and easy way to do this (hopefully by passing some arguments to bjam).
Thanks in advance.

Alexey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100227/40e36441/attachment.html>

From alexey.akimov85 at gmail.com  Sun Feb 28 16:47:55 2010
From: alexey.akimov85 at gmail.com (Alexey Akimov)
Date: Sun, 28 Feb 2010 09:47:55 -0600
Subject: [C++-sig] profiling python extension
In-Reply-To: <47466.76886.qm@web111404.mail.gq1.yahoo.com>
References: <382d227f1002271952p25858f1eyeb1c495bd282f2b@mail.gmail.com>
	<47466.76886.qm@web111404.mail.gq1.yahoo.com>
Message-ID: <382d227f1002280747v44ef247cp6f2516f24b200469@mail.gmail.com>

Thank you, Ralf.
All that sounds pretty promising. I took a look on their web-site - indeed
it does what I whant. However, I am still woundering if there any options to
do this with bjam? Moreover I was successful to compile and run the
extension with variant=profile option and I suspect that there should be
some information generated, but I do not know how to see it. So if someone
here knows how to do this - that should be very valuable knowledge. Anyways
it looks that up to now the oprofile - is my only option.

Best wishes
Alexey

2010/2/27 Ralf W. Grosse-Kunstleve <rwgk at yahoo.com>

> I usually use oprofile. Typically, I write a small piece of python to call
> the
> extension multiple times so it runs the critical code for about a minute
> (10
> seconds will probably do, too), then the extension code is the only thing
> showing up near the top of the oprofile stats.
> It is important that nothing else is running on the machine when you run
> your extension for profiling, since oprofile is a kernel-level tool and
> sees
> the entire system.
> Last time I looked at the docs (ca. 5 years ago...) you had to be root for
> "opcontrol --start". I'm not sure if that's still a requirement.
> The commands I use are:
>
> opcontrol --reset; opcontrol --start # root
> # run as usual (not root)
> opcontrol --shutdown # root
> opreport -l > report
> opannotate --source > annotated
> grep '^[ 0-9][ 0-9][ 0-9][ 0-9][ 0-9][0-9] ' annotated | sort -n
>
> You may need "yum install oprofile" or similar.
>
> ------------------------------
> *From:* Alexey Akimov <alexey.akimov85 at gmail.com>
> *To:* Development of Python/C++ integration <cplusplus-sig at python.org>
> *Sent:* Sat, February 27, 2010 7:52:17 PM
> *Subject:* [C++-sig] profiling python extension
>
> Dear all,
>
> Does anyone know how one may profile a python extention? There is a bunch
> of literature on profiling of pure binaries (programs and even libraries),
> pure python code, etc. However the profiling of a python module from python
> script seems a tricky procedure.
> What I tried is to pass variant=profile to bjam. Everything works fine and
> I get .so library corresponding to my module (placed into /profile
> sub-directory of this particular build). When I execute the python script
> invoking my module's functions (performance of those functions is most
> interesting for me) there is no any output generated (something similar to
> gmon.out). So that is basically my question - how can I retrieve the
> profiling information about my extension? Of course I can compile a normal
> executable using the functions in the module and then use a standard
> approach (and of course gprof), but I hope that there should be more
> straightforward and easy way to do this (hopefully by passing some arguments
> to bjam).
> Thanks in advance.
>
> Alexey
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100228/e39a4131/attachment.html>

From nitroamos at gmail.com  Sun Feb 28 17:40:11 2010
From: nitroamos at gmail.com (Amos Anderson)
Date: Sun, 28 Feb 2010 08:40:11 -0800
Subject: [C++-sig] profiling python extension
Message-ID: <9e910971002280840t71c42ae1xf4c87cafc70ff2c5@mail.gmail.com>

> Date: Sat, 27 Feb 2010 21:52:17 -0600
> From: Alexey Akimov <alexey.akimov85 at gmail.com>
> To: "Development of Python/C++ integration" <cplusplus-sig at python.org>
> Subject: [C++-sig] profiling python extension
> Message-ID:
> ? ? ? ?<382d227f1002271952p25858f1eyeb1c495bd282f2b at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Dear all,
>
> Does anyone know how one may profile a python extention? There is a bunch of
> literature on profiling of pure binaries (programs and even libraries), pure
> python code, etc. However the profiling of a python module from python
> script seems a tricky procedure.
> What I tried is to pass variant=profile to bjam. Everything works fine and I
> get .so library corresponding to my module (placed into /profile
> sub-directory of this particular build). When I execute the python script
> invoking my module's functions (performance of those functions is most
> interesting for me) there is no any output generated (something similar to
> gmon.out). So that is basically my question - how can I retrieve the
> profiling information about my extension? Of course I can compile a normal
> executable using the functions in the module and then use a standard
> approach (and of course gprof), but I hope that there should be more
> straightforward and easy way to do this (hopefully by passing some arguments
> to bjam).
> Thanks in advance.
>
> Alexey


If you're using OSX, then the Shark tool seems to work really well,
and is really easy to use. I does not require that you compile in -pg
mode.

http://developer.apple.com/tools/shark_optimize.html

Amos.

From alexey.akimov85 at gmail.com  Sun Feb 28 17:58:45 2010
From: alexey.akimov85 at gmail.com (Alexey Akimov)
Date: Sun, 28 Feb 2010 10:58:45 -0600
Subject: [C++-sig] profiling python extension
In-Reply-To: <9e910971002280840t71c42ae1xf4c87cafc70ff2c5@mail.gmail.com>
References: <9e910971002280840t71c42ae1xf4c87cafc70ff2c5@mail.gmail.com>
Message-ID: <382d227f1002280858u6b98aa6ala00932a64aa0acbe@mail.gmail.com>

Thank you, Amos
I forget to mention - I am working Linux, so the shark tool probably will
not be suitable for me. But anyway thanks for you reply.

Best wishes
Alexey

2010/2/28 Amos Anderson <nitroamos at gmail.com>

> > Date: Sat, 27 Feb 2010 21:52:17 -0600
> > From: Alexey Akimov <alexey.akimov85 at gmail.com>
> > To: "Development of Python/C++ integration" <cplusplus-sig at python.org>
> > Subject: [C++-sig] profiling python extension
> > Message-ID:
> >        <382d227f1002271952p25858f1eyeb1c495bd282f2b at mail.gmail.com>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Dear all,
> >
> > Does anyone know how one may profile a python extention? There is a bunch
> of
> > literature on profiling of pure binaries (programs and even libraries),
> pure
> > python code, etc. However the profiling of a python module from python
> > script seems a tricky procedure.
> > What I tried is to pass variant=profile to bjam. Everything works fine
> and I
> > get .so library corresponding to my module (placed into /profile
> > sub-directory of this particular build). When I execute the python script
> > invoking my module's functions (performance of those functions is most
> > interesting for me) there is no any output generated (something similar
> to
> > gmon.out). So that is basically my question - how can I retrieve the
> > profiling information about my extension? Of course I can compile a
> normal
> > executable using the functions in the module and then use a standard
> > approach (and of course gprof), but I hope that there should be more
> > straightforward and easy way to do this (hopefully by passing some
> arguments
> > to bjam).
> > Thanks in advance.
> >
> > Alexey
>
>
> If you're using OSX, then the Shark tool seems to work really well,
> and is really easy to use. I does not require that you compile in -pg
> mode.
>
> http://developer.apple.com/tools/shark_optimize.html
>
> Amos.
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100228/74f1570a/attachment.html>

From troy at resophonic.com  Sun Feb 28 18:11:17 2010
From: troy at resophonic.com (troy d. straszheim)
Date: Sun, 28 Feb 2010 12:11:17 -0500
Subject: [C++-sig] profiling python extension
In-Reply-To: <382d227f1002280858u6b98aa6ala00932a64aa0acbe@mail.gmail.com>
References: <9e910971002280840t71c42ae1xf4c87cafc70ff2c5@mail.gmail.com>
	<382d227f1002280858u6b98aa6ala00932a64aa0acbe@mail.gmail.com>
Message-ID: <4B8AA3B5.1070101@resophonic.com>

Alexey Akimov wrote:
> Thank you, Amos
> I forget to mention - I am working Linux, so the shark tool probably 
> will not be suitable for me. But anyway thanks for you reply.
> 

Or this:

valgrind --tool=callgrind mypythonscript.py
kcachegrind

-t

From alexey.akimov85 at gmail.com  Sun Feb 28 19:35:22 2010
From: alexey.akimov85 at gmail.com (Alexey Akimov)
Date: Sun, 28 Feb 2010 12:35:22 -0600
Subject: [C++-sig] profiling python extension
In-Reply-To: <4B8AA3B5.1070101@resophonic.com>
References: <9e910971002280840t71c42ae1xf4c87cafc70ff2c5@mail.gmail.com>
	<382d227f1002280858u6b98aa6ala00932a64aa0acbe@mail.gmail.com>
	<4B8AA3B5.1070101@resophonic.com>
Message-ID: <382d227f1002281035u7b6b9aebp1f23809d60f98f24@mail.gmail.com>

Thank you, Troy

I'll try the valgrind tool as well (in addition to oprofile).

Best wishes
Alexey

2010/2/28 troy d. straszheim <troy at resophonic.com>

> Alexey Akimov wrote:
>
>> Thank you, Amos
>> I forget to mention - I am working Linux, so the shark tool probably will
>> not be suitable for me. But anyway thanks for you reply.
>>
>>
> Or this:
>
> valgrind --tool=callgrind mypythonscript.py
> kcachegrind
>
> -t
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100228/4369794d/attachment.html>

From j.reid at mail.cryst.bbk.ac.uk  Sun Feb 28 19:35:28 2010
From: j.reid at mail.cryst.bbk.ac.uk (John Reid)
Date: Sun, 28 Feb 2010 18:35:28 +0000
Subject: [C++-sig] profiling python extension
In-Reply-To: <382d227f1002271952p25858f1eyeb1c495bd282f2b@mail.gmail.com>
References: <382d227f1002271952p25858f1eyeb1c495bd282f2b@mail.gmail.com>
Message-ID: <hmed1g$4kl$1@dough.gmane.org>

Alexey Akimov wrote:
> Does anyone know how one may profile a python extention?

I've used the google profiler on Linux successfully in the past. It is 
quite straightforward to set up inside an extension.

http://goog-perftools.sourceforge.net/doc/cpu_profiler.html

John.


From alexey.akimov85 at gmail.com  Sun Feb 28 22:33:02 2010
From: alexey.akimov85 at gmail.com (Alexey Akimov)
Date: Sun, 28 Feb 2010 15:33:02 -0600
Subject: [C++-sig] profiling python extension
In-Reply-To: <hmed1g$4kl$1@dough.gmane.org>
References: <382d227f1002271952p25858f1eyeb1c495bd282f2b@mail.gmail.com>
	<hmed1g$4kl$1@dough.gmane.org>
Message-ID: <382d227f1002281333g534595e5i6d7408fe129a4730@mail.gmail.com>

Thank you John,

I just installed the tool you suggested and tried to use it. However I do
not see any output. Could you give some more detailes of how the profiling
process with the google-profiler should look like?
What I do is:

1) bjam variant=profile toolset=gcc cxxflags=-fno-for-scope
linkflags=-lprofiler test_ext

2) env CPUPROFILE=/path/to/my/dir/out.prof python ./mypythonscript.py

This runs my python script which invokes the extension. However it does not
produce any output files (which i expect to be out.prof in my working
directory). What should I do in order to see the profiling information?


By the way, by this time I've tried other two tools suggested. Here are my
feelings:

1) oprofile - looks and works good. it is fast enough and produces much
useful information, there is a bunch of options to play with. The only
inconvenience is that I need to start and finish it only being a root. This
means I either have to switch back and forth oftenly or do all profiling
(and development as a root).
2) valgrind - also is pretty convenient tool and produces a lot of
information (the raw output loooks quite difficult to understand, but i
guess there is a play around options). It gives you many options of
profiling. The only bad thing - it is very slow.
3) google-profiling - is kind a promising for my purposes, but by this time
i still have not managed how to see the output information.

Best wishes
Alexey

2010/2/28 John Reid <j.reid at mail.cryst.bbk.ac.uk>

> Alexey Akimov wrote:
>
>> Does anyone know how one may profile a python extention?
>>
>
> I've used the google profiler on Linux successfully in the past. It is
> quite straightforward to set up inside an extension.
>
> http://goog-perftools.sourceforge.net/doc/cpu_profiler.html
>
> John.
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20100228/487f10e6/attachment.html>

From troy at resophonic.com  Sun Feb 28 22:40:05 2010
From: troy at resophonic.com (troy d. straszheim)
Date: Sun, 28 Feb 2010 16:40:05 -0500
Subject: [C++-sig] profiling python extension
In-Reply-To: <382d227f1002281333g534595e5i6d7408fe129a4730@mail.gmail.com>
References: <382d227f1002271952p25858f1eyeb1c495bd282f2b@mail.gmail.com>	<hmed1g$4kl$1@dough.gmane.org>
	<382d227f1002281333g534595e5i6d7408fe129a4730@mail.gmail.com>
Message-ID: <4B8AE2B5.8060109@resophonic.com>

Alexey Akimov wrote:
> 2) valgrind - also is pretty convenient tool and produces a lot of 
> information (the raw output loooks quite difficult to understand, but i 
> guess there is a play around options). It gives you many options of 
> profiling. 

use kcachegrind to inspect the call graph.