Pipes, Sockets and Threads

[David Bolen]

rpevans at my-deja.com writes:

> I have extended Python with some purpose built functions, written in C
> which communicate with a server process, via named pipes.  Initially I
> called one of these functions (mylib.fred ) from main to initiate
> communications with the server. Once stablished I then started a thread,
> again a mylib function, to monitor the pipe connection set up
> previously.  This is does but locks out the rest of the system. Only
> when the thread receives data from the server does the rest of the
> system run momentarily before again hanging. My first line of attack was
> to put the function which initiated the comms in a separate thread so
> divorcing the comms from main completely (I hoped). Now I have the
> problem of communicating data ( pipe/socket fds) from my first thread
> back to main and then to the second thread which inplements the select .

I think you're going to have to be clearer here just what sort of
threads (Python or native) you're talking about in each portion of the
above paragraph, because the answers are different in either case.

If, for example, your mylib module creates an internal native thread
to handle I/O to the named pipe, buffering it up until the Python
process wants to query it, you shouldn't have any problem at all.
Once the native thread is begun, it runs independently of Python.  You
will, of course, have to provide some sort of protection mechanism
(semaphores, critical sections, etc..) over common resources such as
shared buffers.

However, if the threads you are talking about are Python threads, then
it sounds like you aren't relinquishing the global interpreter lock
around blocking I/O in your extension module.  Python, although it
supports threading, can only really have one interpreter native thread
running at one time.  So even if you create multiple Python threads,
if one of them calls into an extension module that then blocks (such
as on I/O to a named pipe), it will block the Python interpreter in
general.

There are macros that you can use within your extension module to
relinquish the global interpreter lock around such blocking I/O or
other long-running native tasks (regaining it after the process is
completed), which will permit the other Python "threads" to continue
running in the interpreter.  The use of these macros and the issue of
the global interpreter lock is covered in the Python/C API
documentation (see seciton 8.1, although I agree it's not unbelievably
detailed).

> (...)
> Secondly how do I transfer variables from the C extension library to and
> from the Python module.  I asume I can't return a value from a thread in
> the argument list. I have tried declaring a variable as global in python
> and external in C without much (ie any succcess ).

Extension functions accept arguments as Python objects (which can be
parsed with PyArg_ParseTuple() for example) and returns results also
as Python objects (often built with Py_BuildValue() or returned from
some other Python C-API function).  This is covered in the extending
and embedding Python documentation.

> Could anyone tell me how I can implement complete asynchronicity between
> my mian module and the communication threads. Is there a site with some
> example Python threaded code. The library reference is quite sparse in
> this area. The books I have bought (O Reilly) do not mention threaded
> architecture. Is there a book which covers this?

Have you read the Python/C API documentation?  It should answer some
of these questions.  As mentioned above, there are probably two main
approaches that you might take.  You could either build internal
threads into your extension module that have nothing to do with Python
directly, but simply ensure your application remains responsive while
the Python threads are doing their thing.  The other approach is to
actually depend on the Python threads for the shared operations and
just support the interpreter lock within your extension functions to
ensure that you don't block other Python threads out.

Which is more appropriate depends on your application - the builtin
Python modules pretty much all do the latter (I believe), but I often
find the former useful in extension modules - particularly when the
reason for the extra threads is just to overlap I/O and remain
responsive to some external device or network connection.

--
-- David
-- 
/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/