UnicodeDecodeError having fetch web page

[Philip Semanchuk]

On May 25, 2010, at 4:00 PM, Barry wrote:

> On 25 Maj, 21:39, Philip Semanchuk <phi... at semanchuk.com> wrote:
>> On May 25, 2010, at 3:13 PM, Barry wrote:
>>
>>
>>
>>> Hi,
>>
>>> The code below is giving me the error:
>>
>>> Traceback (most recent call last):
>>>  File "C:\Users\Administratör\Desktop\test.py", line 4, in <module>
>>> UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in  
>>> position 1:
>>> unexpected code byte
>>
>>> What am i doing wrong?
>>
>>> Thanks,
>>
>>> Barry
>>
>>> request = urllib.request.Request(url='http://en.wiktionary.org/wiki/
>>> baby',headers={'User-Agent':'Mozilla/5.0 (X11; U; Linux i686) Gecko/
>>> 20071127 Firefox/2.0.0.11'} )
>>
>>> response = urllib.request.urlopen(request)
>>> html = response.read().decode('utf-8')
>>
>> Well, for starters you're assuming that the response content is in
>> UTF-8. You need to examine the Content-Type header to see what the
>> encoding is. If it's not UTF-8, there's your problem.
>>
>> HTH
>> P
>
> The content type is utf-8:
>
> Date: Wed, 19 May 2010 19:17:39 GMT
> Server: Apache
> Cache-Control: private, s-maxage=0, max-age=0, must-revalidate
> Content-Language: en
> Vary: Accept-Encoding,Cookie
> Last-Modified: Wed, 19 May 2010 10:10:34 GMT
> Content-Encoding: gzip
> Content-Length: 25247
> Content-Type: text/html; charset=utf-8
> X-Cache: HIT from sq61.wikimedia.org
> X-Cache-Lookup: HIT from sq61.wikimedia.org:3128
> Age: 520549
> X-Cache: HIT from amssq32.esams.wikimedia.org
> X-Cache-Lookup: HIT from amssq32.esams.wikimedia.org:3128
> X-Cache: MISS from amssq37.esams.wikimedia.org
> X-Cache-Lookup: MISS from amssq37.esams.wikimedia.org:80
> Connection: close

Looks like the content is gzipped. Have you unzipped it? Also, from  
where are you getting those headers? The server might well send  
different headers to your browser than to a urllib request.

Have you examined the raw content in a hex editor on in the debugger?  
That would probably answer a lot of questions.


> Can it be that the page is corrupt?

Of course that's always possible, but personally whenever I have to  
decide whether bits are being flipped at random or my code is buggy,  
it's almost always the latter.

> If so, how can I make the best of the situation?

Depends on what you're trying to accomplish.



bye
Philip