Need urllib.urlretrieve and urllib2.OpenerDirector together

Juanlu_001 juanlu001 at gmail.com
Sun Dec 26 18:03:29 EST 2010


Hello everyone. I'm writing a script in Python 2.7 which uses a
urllib2.OpenerDirector instance via urllib2.build_opener() to take
advantage of the urllib2.HTTPCookieProcessor class, because I need to
store and re-send the cookies I get:

opener =
urllib2.build_opener(urllib2.HTTPCookieProcessor(cookielib.CookieJar()))

However, after making several requests and moving the cookies around,
eventually I need to retrieve a list of URLs. I wanted to use
urllib.urlretrieve() because I read it downloads the file in chunks,
but I cannot because I need to carry my cookies on the request and
urllib.urlretrieve() uses a urllib.URLOpener, which doesn't have
support for cookie handlers like OpenerDirector has.

I want to know the reason of this strange way of splitting
functionality, and how can I achieve my goal.

Thank you in advance!



More information about the Python-list mailing list