urllib2 - closing sockets

Anand Pillai pythonguy at Hotpop.com
Thu Jun 3 04:41:49 EDT 2004


I recently noted that urllib2.urlopen(...) for http:// urls
does not make an explicit call to close the underlying 
HTTPConnection socket once the data from the socket is read.

This might not be required since the garbage collector will
close & collect open sockets that are not closed, but it might
cause the system to run out of socket memory if there are 
multiple threads, each opening a socket and the gc not running
in between.

This specifically happens in my HarvestMan program which uses
multiple threads to achieve fast offline web downloads.

A patch to fix this in urllib2.py would be nice.

Thanks

-Anand



More information about the Python-list mailing list