extra gibberish interspersed into urllib2 output

[Andrew Dalke]

Dan Stromberg wrote:
> I'm attempting to retrieve some data from an http server using basic auth
> via python 2.3 with the urllib2 and cookielib modules.
> 
> I'm finding that I'm getting the data I need, but unfortunately, there are
> small bits of gibberish interspersed in it, rendering the data difficult
> to use at best. 
   ..
> Has anyone seen this before?  Is it premature to start using urllib2 from
> python 2.4?  Is it a bad idea to use this 2.4 module on python 2.3?

I've been using urllib2 under Python 2.4 and not seen
problems.  The code hasn't changed much in years, that
I know.

Have you tried doing the request manually?  That is,

%telnet machine 80
GET /asdf HTTP/1.0
  ..put cookie and auth information here..


Doing that would help show the problem is coming from
upstream of Python (or that it's in Python).

To go real hard core you could get ethereal or some
other network sniffer and watch exactly what Python
does.  That's easier in some sense because you don't
need to figure out what to send for the request headers.


				Andrew
				dalke at dalkescientific.com