Threading a lengthy C function

[sdd]

Leo Breebaart wrote:
> ...
> If, however, I use the kind of run() method that I *really* need:
>     def run(self):
>       retvalue = mypackage.sleep(50)
> 
> where mypackage.sleep() is in fact a SWIG-generated wrapper
> around the C library sleep() function the entire application
> will *still* lock up completely until this task is done.
As you suspect, this is the GIL at work.

> I was wondering if anybody could point me in the direction of
> a solution or workaround to my actual problem.
The question is what you want to do.  It seems you want the
python code to run in a thread in parallel with the mypackage.
Do you want the code in mypackage to be able to run in parallel
with itself?  Lots of C code assumes it is the only thread
manipulating its variables, but some doesn't.  Typically
you will need a "mypackage" lock.  Your C interface code will
have to do something conceptually like:
     release_the_GIL();                /*A*/
       acquire_the_mypackage_lock();   /*B*/
          perform mypackage.whatever();/*C*/
       release_the_mypackage_lock();   /*D*/
     acquire_the_GIL();                /*E*/

_BUT_ after you "release the GIL" (between points A and E) you
may no longer talk to python.  You cannot call conversion
functions, python memory allocators, etc. because you might be
running during the garbage collector, or an awkwardly timed
call to the same or a similar memory allocator.

For similar reasons, you cannot even read memory inside python
objects that is mutable, nor should you access memory inside
immutable python objects where you don't "hold a refcnt."

There are similar constraints on data from mypackage.  You
need to think long and hard about the orders of both the
"drop the GIL" and "grab the mypackage lock" prelude to
calling whatever, as well as a long similar thought about
"grab the GIL" and "drop the mypackage lock" postlude to
the call.

The simplest way to think about this stuff is that with GIL
you may read/write python memory.  With the whatever lock,
you may read/write whatever memory safely.  Only if you
hold both locks can you move information between the two.
Think of acquire as a wait, and release as a potentially
very slow operation (it may immediately go do the work
that the lock was holding up).  The potentially slow nature
of both acquire and release is what makes these decisions
tough.

If you didn't really care about the latencies at the lock
operations, this would be easiest:

     acquire_the_mypackage_lock();     /*AA*/
       release_the_GIL();              /*BB*/
          perform mypackage.whatever();/*CC*/
       acquire_the_GIL();              /*DD*/
     release_the_mypackage_lock();     /*EE*/

At point AA, you move data (inputs) from python to mypackage
memory, At point DD, you move data (outputs) from mypackage
to python memory.

With all of the above in mind, the python version you are
working with makes a big difference; the the GIL re-acquire
is being simplified.  2.3 is simpler than earlier versions.
A good strategy is to grab recent python source that behaves
in a way you like, understand how it does its locks, and copy
it.

-Scott David Daniels
Scott.Daniels at Acm.Org