[Python-checkins] python/dist/src/Lib urllib2.py,1.73,1.74

[jhylton at users.sourceforge.net]

Update of /cvsroot/python/python/dist/src/Lib
In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv27132

Modified Files:
	urllib2.py 
Log Message:
Fix urllib2.urlopen() handling of chunked content encoding.

The change to use the newer httplib interface admitted the possibility
that we'd get an HTTP/1.1 chunked response, but the code didn't handle
it correctly.  The raw socket object can't be pass to addinfourl(),
because it would read the undecoded response.  Instead, addinfourl()
must call HTTPResponse.read(), which will handle the decoding.

One extra wrinkle is that the HTTPReponse object can't be passed to
addinfourl() either, because it doesn't implement readline() or
readlines().  As a quick hack, use socket._fileobject(), which
implements those methods on top of a read buffer.  (suggested by mwh)

Finally, add some tests based on test_urllibnet.

Thanks to Andrew Sawyers for originally reporting the chunked problem.


Index: urllib2.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/urllib2.py,v
retrieving revision 1.73
retrieving revision 1.74
diff -C2 -d -r1.73 -r1.74
*** urllib2.py	3 Aug 2004 12:59:41 -0000	1.73
--- urllib2.py	7 Aug 2004 17:40:50 -0000	1.74
***************
*** 998,1003 ****
  
          # Pick apart the HTTPResponse object to get the addinfourl
!         # object initialized properly
!         resp = addinfourl(r.fp, r.msg, req.get_full_url())
          resp.code = r.status
          resp.msg = r.reason
--- 998,1015 ----
  
          # Pick apart the HTTPResponse object to get the addinfourl
!         # object initialized properly.
! 
!         # Wrap the HTTPResponse object in socket's file object adapter
!         # for Windows.  That adapter calls recv(), so delegate recv()
!         # to read().  This weird wrapping allows the returned object to
!         # have readline() and readlines() methods.
!         
!         # XXX It might be better to extract the read buffering code
!         # out of socket._fileobject() and into a base class.
!         
!         r.recv = r.read
!         fp = socket._fileobject(r)
!         
!         resp = addinfourl(fp, r.msg, req.get_full_url())
          resp.code = r.status
          resp.msg = r.reason