[C++-sig] Am I crazy or is it my computer?

[Matthew Scouten]

On 11/1/07, Niall Douglas <s_sourceforge at nedprod.com> wrote:
>
> On 31 Oct 2007 at 15:11, Matthew Scouten wrote:
>
> > Ok, I just found this:
> > http://www.boost.org/libs/python/doc/v2/faq.html#threadsupport
> >
> > I seems to be saying that I can not do what I want with out patching
> > Boost::Python (or does this only apply to multiple interpretors? )
> > Is this still true (please tell me it's been fixed)?
> > if so, where can I get this patch and how do I apply it?
>
> Follow the link to TnFOX given in the FAQ. Then study FXPython -
> heavily - and with a very fine toothed comb.
>
> > I asked about multi-threading and boost python in on this list a while
> ago,
> > and got no answer. I took this a a "go ahead, no problems". If I am
> being an
> > idiot, could you tell me so? Just flame away at me so I know I've been
> > stupid.
>
> I remember your question, and I didn't answer because the FAQ answers
> and I don't have the time to answer FAQ answered questions. I think
> the Boost.Python homepage could do with a large flashing neon sign
> saying "CHECK THE FAQ BEFORE ASKING A QUESTION" right before "SEARCH
> THE C++ SIG MAILING LIST ARCHIVES BEFORE ASKING A QUESTION".
>
> Had you done either, you would have known the answer.
>
> Now with the berating being done, I must express my sympathies for
> your task. I wrapped TnFOX for Python using Boost.Python and it was
> extremely painful indeed. It can be done, but expect at least 200
> hours of programmer time to wrap your head around it. Once you know
> what you're doing, it becomes quite easy.
>
> > A lot of people here actually know what they are doing. I do not
> > have that luxury.  I am finding boost python extremely complex and
> > non-intuitive.  There is also a lack of good documentation (example code
> is
> > nice to have, but does not constitute proper documentation all by
> itself).
>
> Actually, the docs are pretty good. I would have cut out my own
> tongue to admit that a few years ago, but the problem is not the
> documentation - it's the programmer thinking in old style C++ ways.
>
> Once you wrap your head around Boost.Python, it becomes quite
> intuitive. I can now happilly plug around its internals and do
> anything I like fairly confidently. It's actually quite a simple &
> intuitive design though I do wish there were more usage examples out
> there - and also I wish more interfaces were publicly defined &
> solid.
>
> Now I have designed the machinery needed to get threaded C++ and
> Python working. It took me months, and it's not pretty at all and
> rather easy to break if you modify it. But copying me would be the
> easiest by far.
>
> BTW I reckon your problems stem from the GIL not only being a lock
> but also being a state variable for Python. This state varies,
> unpredictably, during and between locks whenever you touch Python at
> all (including via any other thread). Touching BPL in any way usually
> means touching Python, so basically dragons abound here.
>
> This will explain the often erratic & weird ordering of actions in
> FXPython in TnFOX - I worked it out after many hours in the debugger
> watching Python, C++, and Boost.Python interact. I suggest you copy
> my form closely in your endeavours.
>
> Good luck!
>
> Cheers,
> Niall



I have downloaded your code and had a look at it. I have a few questions.

First of all I do not have and do not need multiple interpretors(thank
goodness). I am not sure what parts of what you did are for multi
interpretors and what parts are for simple multi threading with one
interpretor. I noticed that you did not use the PyGILState_Ensure &
PyGILState_Release pair. This makes sense, because they is meant for a
single interpretor environment.

What I do not understand is: why they would not be sufficient in my
environment. from the information at:
http://svn.python.org/projects/peps/trunk/pep-0311.txt
and
http://docs.python.org/api/threads.html

I can safely say that I am certain that they would be sufficient in a pure
C/Python environment. I am calling _Ensure before touching any boost python,
and not calling _Release until I am done with all boost python calls in a
given callback. I have posted my example before:

void DataConsumer_wrapper::OnSampleCallback( int a1, int a2)
{
    {callback call;
        if (override func = this->get_override("OnSampleCallback"))
        {
            logfunc log("DataConsumer_wrapper::OnSampleCallback", true);
            func(a1,a2 );
            return;
        }
    }// end callback call;
    return DataConsumer::OnSampleCallback(a1,a2);
}

This is a virtual function callback. The "callback call" line is calling
PyGILState_Ensure:

class callback
{
public:
    inline callback()
    {
        state = PyGILState_Ensure();
    }

    inline ~callback()
    {
        PyGILState_Release(state);
    }

private:
    PyGILState_STATE state;
};

So If this was pure C/Python, I have obeyed the rules. My question is:
Is there something in boost::python itself (as opposed to C/Python) that may
be having a problem with this model?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20071102/03630766/attachment.htm>