[Web-SIG] Threading and client-side support

[Ian Bicking]

On Saturday, October 25, 2003, at 07:18 PM, John J Lee wrote:
> First, I should state that I'm almost entirely ignorant of all things
> threads.  Be gentle with me.
>
> What is the current state of thread-safety in the Python standard 
> library
> client-side web code (ie. httplib, urllib, urllib2)?

As far as I know they are threadsafe.

> I ask because my cookies code is currently entirely thread-ignorant, 
> and
> I'm wondering if it should have appropriate thread synchronization -- 
> and
> if so, what problems I'm supposed to be preventing, and how to prevent
> them.

It's all about concurrent access.  For instance, looking at 
ClientCookie, the question would be what would happen when 
ClientCookie.urlopen was called while another ClientCookie.urlopen was 
running.  For instance, in ClientCookie._urllib2_support.urlopen, 
build_opener() can be called twice.  If this is a problem then the code 
isn't threadsafe (i.e., if build_opener() isn't threadsafe then urlopen 
isn't threadsafe).  urlopen() can protect build_opener() with a lock, 
like:

urlopen_lock = threading.Lock()
def urlopen(url, data=None):
     global _opener
     if _opener is None:
         urlopen_lock.acquire()
         try:
             if _opener is None:
                 # it might not be None, because we might have called 
build_opener()
                 # sometime between the first if and acquiring the 
lock...
                 _opener = build_opener()
         finally:
             urlopen_lock.release()
      return _opener.open(url, data)

There's a little more complexity there so that you don't have to 
acquire the lock every time you call urlopen().  _opener.open() still 
has to be threadsafe at this point (and you'll definitely want it to be 
threadsafe, so requests don't have to be done serially).

Where you have to do this sort of thing depends on what parts of the 
system are exposed so that they can be used concurrently.

--
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org