cookielib and urllib2: thread-safe?

[John J. Lee]

Alex Hunsley <lard at tardis.ed.ac.molar.uk> writes:

> I'm writing a test script in python for pulling web pages from a web
> server using urllib2 and cookielib. Since the main thing I am testing
> is what happens when concurrent requests are made to the web server, I
> need to make several requests concurrently, which I'll do from
> different threads in my python script. So the important question is:
> are cookielib and urllib2 thread safe? Are there any precautions that
> apply to using these libs in a multi-threaded context?

urllib2: For HTTP, yes, AFAIK.  For other protocols, eg. FTP, perhaps
not.

cookielib: No.  It's currently thread-broken, simply because I've
never had reason to do threaded stuff with it.  There is thread
synchronization code in there, but I have little doubt that it's
broken, because it's untested (I should have removed the
synchronization code entirely, in fact, since nobody volunteered to
test & fix before release: unfortunately the first beta caught me by
surprise...).

I won't be making it threadsafe, so don't wait for me to do it.  I'm
happy to help others do so.  Nothing especially hard for somebody with
plenty of thread experience to do, AFAIK.

I had thought part of a patch had gone in which stated this
thread-unsafety very prominently in the cookielib module docs, but it
seems that never happened, or at least not in time for 2.4.0 -- eek!


John