[C++-sig] boost::python and threads
Renato Araujo
renatox at gmail.com
Wed Jul 8 19:23:53 CEST 2009
Ok I made 2 modifications and I got this working here.
Try this.
51,54c50
< virtual void onTick() {
< if (object o = get_override("onTick"))
< o();
< }
---
> virtual void onTick() { get_override("onTick")(); }
60d55
< PyEval_InitThreads();
65a61
>
BR
On Wed, Jul 8, 2009 at 12:48 PM, Paul Scruby<paul at gingernut.tv> wrote:
> Hi Renato,
>
> Okay, I have installed Python 2.4.2 with gcc 4.1.2 on Linux and it's
> crashing in the same place as on Solaris.
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x41074940 (LWP 12004)]
> 0x000000307740c850 in sem_post () from /lib64/libpthread.so.0
> (gdb) where
> #0 0x000000307740c850 in sem_post () from /lib64/libpthread.so.0
> #1 0x000000308b2b78e9 in PyThread_release_lock ()
> from /usr/lib64/libpython2.4.so.1.0
> #2 0x00002b46531d9f82 in Ticker::operator() (this=0xbd0f478) at
> ticker.cc:28
> #3 0x00002b46531d9fb0 in boost::detail::thread_data<Ticker>::run (
> this=0xbd0f370)
> at /opt/atm/include/boost-1_39/boost/thread/detail/thread.hpp:56
> #4 0x00002b46533ee14b in thread_proxy ()
> from /opt/atm/lib64/libboost_thread-gcc41-mt-1_39.so.1.39.0
> #5 0x0000003077406367 in start_thread () from /lib64/libpthread.so.0
> #6 0x00000030768d2f7d in clone () from /lib64/libc.so.6
>
> With the global interpret lock added my code now looks like this...
>
> #include <boost/python.hpp>
> #include <boost/thread/thread.hpp>
> #include <boost/thread/xtime.hpp>
> using namespace boost::python;
>
> class Ticker
> : public wrapper<Ticker>
> {
> private:
> bool run_;
> volatile bool * running_;
> boost::thread * thread_;
> boost::xtime xt_;
> PyGILState_STATE state_;
> public:
> Ticker() :running_(&run_) { *running_ = false; }
>
> void operator()()
> {
> while (*running_)
> {
> boost::xtime_get(&xt_, boost::TIME_UTC);
> ++xt_.sec;
> boost::thread::sleep(xt_);
> state_ = PyGILState_Ensure();
> onTick();
> PyGILState_Release(state_);
> }
> }
>
> void run()
> {
> if (*running_ == false)
> {
> *running_ = true;
> thread_ = new boost::thread(*this);
> }
> }
>
> void stop()
> {
> if (*running_ == true)
> {
> *running_ = false;
> thread_->join();
> delete thread_;
> }
> }
>
> virtual void onTick() { get_override("onTick")(); }
> void default_onTick() {}
> };
>
> BOOST_PYTHON_MODULE(tick)
> {
> class_<Ticker, boost::noncopyable> ("Ticker")
> .def("run", &Ticker::run)
> .def("stop", &Ticker::stop)
> .def("onTick", &Ticker::default_onTick);
> }
>
>
> Thanks again,
>
> Paul
>
>
> "Renato Araujo" <renatox at gmail.com> wrote in message
> news:95291a80907060619u5bff0dcey56947d1ac848cc8e at mail.gmail.com...
> I'm using gcc/linux and python >= 2.4 and works fine for me.
>
>
>
> On Mon, Jul 6, 2009 at 7:39 AM, Paul Scruby<paul at gingernut.tv> wrote:
>> Hello again,
>>
>> Sorry, I spoke too soon. The good news is that wrapping my virtual method
>> calls into Python between PyGILState_Ensure() and PyGILState_Release()
>> fixed
>> the crash under Python2.6.2 on Windows, but not when I tested it again
>> under
>> Python 2.4.4 on Solaris. Under Python 2.4.4 on SolarisSolaris it now calls
>> and exits the virtual method in Python sucessfully, but then crashes when
>> C++ trys to release the global interpretor lock.
>>
>> The call-stack with the Sun C++ 5.9 compiler under Solaris is
>> t at 2 (l at 2) signal SEGV (no mapping at the fault address) in sem_invalid
>> at 0xfec453ed
>> 0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
>> Current function is Ticker::operator()
>> 28 PyGILState_Release(state_);
>> (dbx) where
>> current thread: t at 2
>> [1] sem_invalid(0x0), at 0xfec453ed
>> [2] _sem_post(0x0), at 0xfec454c4
>> [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
>> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
>> [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
>> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
>> [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
>> 0x0, 0x80a3140), at 0xfef43eba
>> [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
>> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
>> [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56
>> in "thread.hpp"
>> [9] thread_proxy(0x810a288), at 0xfea78ce4
>> [10] _thr_setup(0xfe670200), at 0xfee159b9
>> [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
>> 0xfe77ef14), at 0xfee15ca0
>>
>> Do you think it's worth repeating this test using gcc/linux, or do you
>> think
>> that this is just a limitation of using Python with threads?
>>
>> Thanks again,
>>
>> Paul
>>
>>
>> t at 2 (l at 2) signal SEGV (no mapping at the fault address) in sem_invalid at
>> 0xfec453ed
>> 0xfec453ed: sem_invalid+0x0013: movzwl 0x00000006(%eax),%eax
>> Current function is Ticker::operator()
>> 28 PyGILState_Release(state_);
>> (dbx) where
>> current thread: t at 2
>> [1] sem_invalid(0x0), at 0xfec453ed
>> [2] _sem_post(0x0), at 0xfec454c4
>> [3] PyThread_release_lock(0x0, 0xfe77ef2c, 0xfef43eba, 0x80c55c0,
>> 0xfe77ef38, 0xfef441b5), at 0xfef492dc
>> [4] PyEval_ReleaseLock(0x80c55c0, 0xfe77ef38, 0xfef441b5, 0xfeb12aec,
>> 0xfe77ef70, 0xfeafb2e3), at 0xfef27abe
>> [5] PyThreadState_DeleteCurrent(0xfeb12aec, 0xfe77ef70, 0xfeafb2e3, 0x1,
>> 0x0, 0x80a3140), at 0xfef43eba
>> [6] PyGILState_Release(0x1, 0x0), at 0xfef441b5
>> =>[7] Ticker::operator()(this = 0x810a304), line 28 in "ticker.cc"
>> [8] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line 56 in
>> "thread.hpp"
>> [9] thread_proxy(0x810a288), at 0xfea78ce4
>> [10] _thr_setup(0xfe670200), at 0xfee159b9
>> [11] _lwp_start(0xfe77ef08, 0xfec454c4, 0x0, 0xfe77ef54, 0x80c55c0,
>> 0xfe77ef14), at 0xfee15ca0
>>
>>
>>
>> "Paul Scruby" <paul at gingernut.tv> wrote in message
>> news:h2sgic$ad4$1 at ger.gmane.org...
>>> Hiya,
>>>
>>> That's fantastic, all I needed to do was to put PyGILState_Ensure();
>>> before my virtual function calls into python from another thread and my
>>> program no longer crashes. Problem solved, isn't boost::python great!
>>>
>>> Many thanks,
>>>
>>> Paul
>>>
>>>
>>> "Renato Araujo" <renatox at gmail.com> wrote in message
>>> news:95291a80907041315k41b7ad88o32d2111ae8fe1e91 at mail.gmail.com...
>>> Hi Paul
>>>
>>> In my bindings I had a problem like this, to solve I created a simple
>>> class like that:
>>>
>>> class thread_locker
>>> {
>>> thread_locker()
>>> {
>>> if (thread_support::enabled())
>>> m_gstate = PyGILState_Ensure();
>>> }
>>>
>>> ~thread_locker()
>>> {
>>> if (thread_support::enabled())
>>> PyGILState_Release(m_gstate);
>>> }
>>> };
>>>
>>> then in my wrapper virtual implementation I did this:
>>>
>>> ...
>>> void wrapper::virtual_func(..)
>>> {
>>> thread_locker lock;
>>> .. my code ..
>>> }
>>> ....
>>>
>>> this solve my problems with call of virtual functions in thread
>>> enviroment.
>>>
>>> BR
>>>
>>>
>>>
>>>
>>> On Sat, Jul 4, 2009 at 5:03 PM, William Ladwig<wladwig at wdtinc.com> wrote:
>>>> Whoops, I think this problem is a little uglier than I thought, since
>>>> you
>>>> overrode the onTick() function in python with a call to print, which
>>>> needs access to the interpreter in your new thread. See the link Thomas
>>>> posted for dealing with the GIL (along with worrying about any possible
>>>> garbage collection issues). Now I remember why I kept my C++ threads
>>>> isolated from Python stuff....sorry, it's been a while....
>>>>
>>>> Bill
>>>> ________________________________________
>>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com at python.org
>>>> [cplusplus-sig-bounces+wladwig=wdtinc.com at python.org] On Behalf Of
>>>> William Ladwig [wladwig at wdtinc.com]
>>>> Sent: Saturday, July 04, 2009 1:34 PM
>>>> To: Development of Python/C++ integration
>>>> Subject: Re: [C++-sig] boost::python and threads
>>>>
>>>> It looks to me like you have a garbage collection problem going on. If
>>>> you create a wrapped c++ object in python, then python is going to own
>>>> the object and will destroy it when its reference count goes to 0. In
>>>> your python example script at the bottom, you call the Ticker's run()
>>>> function, which from the python point of view, returns quickly and the
>>>> script ends. Python has no idea that you spawned off a new thread from
>>>> the C++ side, so when the script ends, python destroys the object and
>>>> now
>>>> you have a problem. One way that you can check to see if this is what is
>>>> going on is to add this to the bottom of the test script and see if the
>>>> crashes go away:
>>>>
>>>> # Warning, you'll need to kill this script manually
>>>> import time
>>>> while True:
>>>> time.sleep(1)
>>>>
>>>> Generally when I want to fire off a new C++ thread, I hold any objects
>>>> that the thread needs in an auto_ptr (see held type for class wrappers),
>>>> take ownership of them in C++ (see the FAQ in the documentation) and
>>>> start the thread from the C++ side. Or, you can also create C++ wrappers
>>>> which accept shared_ptr arguments, while holding your classes in
>>>> shared_ptrs, and this should handle the reference counting as well.
>>>> Unfortunately, this may require some interface changes to what you have
>>>> already written (or possibly some clever wrapping).
>>>>
>>>> Hope this helps,
>>>> Bill
>>>>
>>>>
>>>> ________________________________________
>>>> From: cplusplus-sig-bounces+wladwig=wdtinc.com at python.org
>>>> [cplusplus-sig-bounces+wladwig=wdtinc.com at python.org] On Behalf Of Paul
>>>> Scruby [paul at gingernut.tv]
>>>> Sent: Friday, July 03, 2009 6:15 AM
>>>> To: cplusplus-sig at python.org
>>>> Subject: [C++-sig] boost::python and threads
>>>>
>>>> I am having some problems using boost::python with boost::thread. I'm
>>>> using
>>>> threads because I want to run some tasks in the background when I'm
>>>> using
>>>> the Python's interactive shell. However, when I use get_override() to
>>>> call
>>>> a Python method from another boost::thread it crashes internally. For
>>>> example:
>>>>
>>>> #include <boost/python.hpp>
>>>> #include <boost/thread/thread.hpp>
>>>> #include <boost/thread/xtime.hpp>
>>>>
>>>> using namespace boost::python;
>>>>
>>>> class Ticker
>>>> : public wrapper<Ticker>
>>>> {
>>>> private:
>>>> bool run_;
>>>> volatile bool * running_;
>>>> boost::thread * thread_;
>>>> boost::xtime xt_;
>>>> public:
>>>> Ticker() : running_(&run_) { *running_ = false; }
>>>>
>>>> void operator()()
>>>> {
>>>> while (*running_)
>>>> {
>>>> boost::xtime_get(&xt_, boost::TIME_UTC);
>>>> ++xt_.sec;
>>>> boost::thread::sleep(xt_);
>>>> onTick();
>>>> }
>>>> }
>>>>
>>>> void run()
>>>> {
>>>> if (*running_ == false)
>>>> {
>>>> *running_ = true;
>>>> thread_ = new boost::thread(*this);
>>>> }
>>>> }
>>>>
>>>> void stop()
>>>> {
>>>> if (*running_ == true)
>>>> {
>>>> *running_ = false;
>>>> thread_->join();
>>>> delete thread_;
>>>> }
>>>> }
>>>>
>>>> virtual void onTick() { get_override("onTick")(); }
>>>> void default_onTick() {}
>>>> };
>>>>
>>>> BOOST_PYTHON_MODULE(tick)
>>>> {
>>>> class_<Ticker, boost::noncopyable> ("Ticker")
>>>> .def("run", &Ticker::run)
>>>> .def("stop", &Ticker::stop)
>>>> .def("onTick", &Ticker::default_onTick);
>>>> }
>>>>
>>>> Here is a test script that which will crash when you import it into
>>>> Python's
>>>> interactive shell.
>>>>
>>>> from tick import Ticker
>>>>
>>>> class MyTicker(Ticker):
>>>> def onTick(self):
>>>> print "Each second"
>>>>
>>>> myticker = MyTicker()
>>>> myticker.run()
>>>>
>>>> I ran this test initially on Python 2.4.4 with the Sun C++ 5.9 compiler
>>>> on
>>>> Solaris and I also tested it using Python 2.6.2 with Visual Studio 2008
>>>> on
>>>> Windows XP.
>>>>
>>>> The call-stack in dbx on Solaris:
>>>>
>>>> >>> t at 2 (l at 2) signal SEGV (no mapping at the fault address) in
>>>> PyErr_Restore at 0xfef38fa1
>>>> 0xfef38fa1: PyErr_Restore+0x0031: movl 0x00000028(%edi),%ecx
>>>> Current function is boost::python::override::operator()
>>>> 99 detail::method_result x(
>>>>
>>>> (dbx) where
>>>> current thread: t at 2
>>>> [1] PyErr_Restore(0x80652fc, 0x80f1220, 0x0, 0xfe77ee90, 0xfef3951e,
>>>> 0x80652fc), at 0xfef38fa1
>>>> [2] PyErr_SetObject(0x80652fc, 0x80f1220), at 0xfef3901e
>>>> [3] PyErr_Format(0x80652fc, 0xfef5c2d8, 0xfef7902c), at 0xfef3951e
>>>> [4] PyObject_Call(0xfef88768, 0x806102c, 0x0), at 0xfeee291a
>>>> [5] PyEval_CallObjectWithKeywords(0xfef88768, 0x806102c, 0x0), at
>>>> 0xfef2bf02
>>>> [6] PyEval_CallFunction(0xfef88768, 0xfeb02004), at 0xfef434c5
>>>> =>[7] boost::python::override::operator()(this = 0xfe77ef30), line 99 in
>>>> "override.hpp"
>>>> [8] Ticker::onTick(this = 0x810a304), line 48 in "ticker.cc"
>>>> [9] Ticker::operator()(this = 0x810a304), line 25 in "ticker.cc"
>>>> [10] boost::detail::thread_data<Ticker>::run(this = 0x810a288), line
>>>> 56 in "thread.hpp"
>>>> [11] thread_proxy(0x810a288), at 0xfea78ce4
>>>> [12] _thr_setup(0xfe670200), at 0xfee159b9
>>>> [13] _lwp_start(0xfe77ef54, 0x80f1220, 0xfe77ee7c, 0xfef3901e,
>>>> 0x80652fc, 0x80f1220), at 0xfee15ca0
>>>>
>>>> The call-stack in Visual Studio 2008:
>>>>
>>>> python26.dll!1e013595()
>>>> [Frames below may be incorrect and/or missing, no symbols loaded for
>>>> python26.dll]
>>>> python26.dll!1e09ee7d()
>>>> > tick.pyd!boost::python::override::operator()() Line 103 + 0x16 bytes
>>>> C++
>>>> 00f3fd64()
>>>> tick.pyd!Ticker::operator()() Line 27 + 0xe bytes C++
>>>> tick.pyd!boost::detail::thread_data<Ticker>::run() Line 57 C++
>>>> tick.pyd!boost::`anonymous namespace'::thread_start_function(void *
>>>> param=0x00245f30) Line 168 C++
>>>> msvcr90d.dll!_callthreadstartex() Line 348 + 0xf bytes C
>>>> msvcr90d.dll!_threadstartex(void * ptd=0x00d46938) Line 331 C
>>>> kernel32.dll!7c80b729()
>>>>
>>>>
>>>> Have a missed a trick using the wrapper, or does boost::python not
>>>> support
>>>> threading?
>>>>
>>>> Many thanks,
>>>>
>>>> Paul
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig at python.org
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig at python.org
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>> _______________________________________________
>>>> Cplusplus-sig mailing list
>>>> Cplusplus-sig at python.org
>>>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>>>
>>>
>>>
>>>
>>> --
>>> Renato Araujo Oliveira Filho
>>
>>
>>
>> _______________________________________________
>> Cplusplus-sig mailing list
>> Cplusplus-sig at python.org
>> http://mail.python.org/mailman/listinfo/cplusplus-sig
>>
>
>
>
> --
> Renato Araujo Oliveira Filho
>
>
>
> _______________________________________________
> Cplusplus-sig mailing list
> Cplusplus-sig at python.org
> http://mail.python.org/mailman/listinfo/cplusplus-sig
>
--
Renato Araujo Oliveira Filho
More information about the Cplusplus-sig
mailing list