Thread-blocking (was: socket.gethostbyname() thread-safe?)

[Johannes Stezenbach]

Michael Ströder <michael at stroeder.com> wrote:
>The whole HTTP serving stuff seems to work just fine. But the
>underlying LDAP libs (OpenLDAP 1.2.x in my case) are known not to be
>thread-safe. Therefore I'm using locks to protect calls into
>python-ldap methods. My problem is that sometimes the whole thing
>blocks. Mainly if the ldap.open() call (creating a new connection)
>does not return or the client aborts the connection.

Sounds like a C code problem. gdb is your friend.
Hint: libpthread redefines some libc functions for thread safety. You
must link ldapmodule with -lpthread. You must also compile with
-D_REENTRANT, *even if ldapmodule itself does not use pthread
functions directly*.

>Two questions:
>- Does the python-ldap module also has to set the global interpreter
>lock as mentioned in the API docs (api/threads.html)?

When the threads are created in Python and call out to a C extension
module, the global interpreter lock is already held by that thread.
Blocking C functions in the extension module should be surrounded
by Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to *release* the
global interpreter lock and allow other threads to run. This is
done correctly in ldapmodule.

>- What is a good practice of exception handling to avoid blocking?
>Should I catch all exceptions to avoid that the thread won't call
>properly thread.exit(). Is this already done by
>SocketServer.ThreadingMixIn?

Calling thread.exit() explicitely is unnecessary.
But see below.

>  def handler(self,*args):
>    req = args[0]
>    try:
>      w2lhandler.HandleHTTPRequest(req.inp,req.out,req.err,req.env)
>    except:
>      # Avoid blocking 
>      pass
>    req.Finish()
>    thread.exit()

Looks fishy. I assume it "blocks" when req.Finish() isn't called
(unterminated fcgi call)? That's what try/finally is for:

def handler(self,*args):
  req = args[0]
  try:
    w2lhandler.HandleHTTPRequest(req.inp,req.out,req.err,req.env)
  finally:
    req.Finish()


HTH,
Johannes