urllib2 and transfer-encoding = chunked

[Nikita the Spider]

In article <1169137502.990783.222150 at 51g2000cwl.googlegroups.com>,
 jdvolz at gmail.com wrote:

> Haha!  My mistake.
> 
> The error is that when a web server is chunking a web page only the
> first chunk appears to be acquired by the urllib2.urlopen call.  If you
> check the headers, there is no 'Content-length' (as expected) and
> instead there is 'transfer-encoding' = 'chunked'.  I am getting about
> the first 30Kb, and then nothing else.
> 
> I don't get a ValueError like described at the following post:

Hi jdvolz,
What error *do* you get? Or is it that no error is raised; you're just 
not getting all of the data? If it is the latter, then the sending 
server might be at fault for not properly following the chunked transfer 
protocol. One way to find out would be to fire up Ethereal and see 
what's coming down the wire. 

> I am having errors which appear to be linked to a previous bug in
> urllib2 (and urllib) for v2.4 and v2.5 of Python.  Has this been fixed?
>  Has anyone established a standard workaround?  I keep finding old
> posts about it, that basically give up and say "well it's a known bug."

Can you give us some pointers to some of these old posts? And tell us 
what version of Python you're using.

-- 
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more