Embeded Multi-Threaded Python: PyEval_InitThread(), PyEval_SaveThread(),...

[Frederic Giacometti]

"Nicolas Duchastel" <nicolas at otelnet.com> wrote in message
news:8e0ac4fb.0109191313.5fef1454 at posting.google.com...
> Thanks for the quick response.... but.....
>
> I actually RTFM-ed it; that's why my question is so detailed.
CYSPE ?? (Translation: Can you speak English ?)
What is this funky RTFM verb ???

> The doc seem to be conflicting in some areas and lacks some important
> information.
>
> For example, Chapter 8 states that Py_Initialized() can be called
> before or after PyEval_InitThreads(); so which is best ? what are the
> implications ? any nuance between these two ways ? any examples ?

This is not conflicting, it just states the order dones not matter; which is
true.

What's missing in the documentation is the indication that
PyEval_InitThreads() creates and initialize the global lock; and thus must
have been called before any real thread operation is engaged.
The code of PyEval_InitThreads is only a couple of lines long. Looking at it
might help you, too.

> Also, chapter 8.1 does indicated 4 different ways to do locking:
>   C_Function_ToCallPythonCode()
>   {
> PyThreadState* save = PyEval_SaveThread(); // LOCK
> PyImport_ImportModule( ...); // load another module
> PyRun_SimpleString(...);      // execute a 1 line of Python code
> PyPObject_CallMethod(....); // execute a full Python method
> PyEval_RestoreThread(save); // UNLOCK

But they don't refer to the same locks.... For instance, there is a global
interpreter lock, and an import lock; these are two different locks.
So, yes, you're locking, but you're not locking the same thing.  Not all
locks are the same, you know :))

>   }
> OR
>   C_Function_ToCallPythonCode()
>   {
> PyEval_AcquireLock(); // LOCK
> PyImport_ImportModule( ...); // load another module
> PyRun_SimpleString(...);      // execute a 1 line of Python code
> PyPObject_CallMethod(....); // execute a full Python method
> PyEval_ReleaseLock(); // UNLOCK
>   }
> OR
>   C_Function_ToCallPythonCode()
>   {
> PyEval_AcquireThread(??); // LOCK
> PyImport_ImportModule( ...); // load another module
> PyRun_SimpleString(...);      // execute a 1 line of Python code
> PyPObject_CallMethod(....); // execute a full Python method
> PyEval_ReleaseThread(??); // UNLOCK

There are two global interpreter lock/thread function/macro sets: one which
is source-compatible with single-threaded Python builds, and another one
which is not.

In addition to this, PyEval_InitThread() does some acrobaties to dynamically
enable/disable thread on python extension modules, so that the same module
can work on mutli-threaded and single-threaded python engine; non of this
being actually documented, of course :))

As resultat des courses, when you invoke binary extensions from the 'python'
command, the PyEval_InitThread() is only executed upon loading a module
requiring threads (e.g. the threads module).
If you create your own binary module which might work on its own thread,
you'll also want to insert a PyEval_InitThread() call in the module
initialisation function.

Voila...

>   }
> OR
>   C_Function_ToCallPythonCode()
>   {
> Py_BEGIN_ALLOW_THREADS
> PyImport_ImportModule( ...); // load another module
> PyRun_SimpleString(...);      // execute a 1 line of Python code
> PyPObject_CallMethod(....); // execute a full Python method
> Py_END_ALLOW_THREADS
>   }
>
> So which one should I use ?

You should use the macros, preferably to explicit function calls, whenever
you can...

> Also, image my 2 distinct thread B and C executing the 1st snipplet of
> code above, if the call to PyObject_CallMethod() is VERY long and
> takes ages,...

Python takes care of it, don't worry :)))

> chapter 8 says
>
>    "In order to support multi-threaded Python programs, the
>     interpreter regularly releases and reacquires the lock --
>     by default, every ten bytecode instructions."
>
> Does that thus mean that a 2nd thread running this 1st snipplet
> above will be interupted in the middle of its work in
> PyObject_CallMethod() and another thread will have a go at it.

That's what thread switching is about, doesn't it ?

> If so, what happens to the ThreadState object when the 2nd thread
> calls PyEval_SaveThread() ? if it doesn't swap it, it will run with
> the 1st thread's ThreadState object. If it does swap it, when this
> 2nd thread will also get interupted, won't the 1st thread get back
> to run and thus it will run with the 2nd thread's object !?
> I mean, I must be missing someting !?

No (or maybe a little imagination :)): Just think of the Python virtual
machine as a single process ressource shared between the native threads.

> Also, what about the things I was asking with regards to the actual OS
> thread's ID ? i.e. does it matter that my code starts in one thread
> and then works from another thread ? do I need to create ThreadState
> objects for each threads ?

?????

> Thus,.. please RTFP (where P is for Posting rather than Manual)

Qu'es aqueo ????

>
> Thanks,

De rien.

> Nicolas

FG