Deflate with urllib2...

[Sam]

For those that are interested, but don't want to bother running the
program themselves, here's the output I get.

Trying:  http://slashdot.org
  http://slashdot.org - Apache/1.3.41 (Unix) mod_perl/1.31-rc4
(deflate)
  Errored out on this site.

Trying:  http://www.hotmail.com
  http://www.hotmail.com - Microsoft-IIS/6.0 (deflate)
  Errored out on this site.

Trying:  http://www.godaddy.com
  http://www.godaddy.com - Microsoft-IIS/6.0 (deflate)
  Errored out on this site.

Trying:  http://www.linux.com
  http://www.linux.com - Apache/2.2.8 (Unix) PHP/5.2.5 (deflate)
  Errored out on this site.

Trying:  http://www.lighttpd.net
  http://www.lighttpd.net - lighttpd/1.5.0 (deflate)
  Errored out on this site.

Trying:  http://www.kenrockwel.com
  http://www.kenrockwel.com - lighttpd (deflate)
  Able to decompress...went from 414 to 744.



On Sep 18, 7:29 pm, Sam <samsli... at gmail.com> wrote:
> On Sep 18, 2:10 pm, "Gabriel Genellina" <gagsl-... at yahoo.com.ar>
> wrote:
>
> > En Tue, 16 Sep 2008 21:58:31 -0300, Sam <samsli... at gmail.com> escribió:
> > The code is correct - try with another server. I tested it with a  
> > LightHTTPd server and worked fine.
>
> Gabriel...
>
> I found a bunch of servers to test it on.  It fails on every server I
> could find (sans one).
>
> Here's the ones it fails on:
> slashdot.org
> hotmail.com
> godaddy.com
> linux.com
> lighttpd.net
>
> I did manage to find one webserver it succeeded on---that is
> kenrockwel.com --- a domain squatter with a typoed domain of one of my
> favorite photographer's websites (the actual website should be
> kenrockwell.com)
>
> This squatter's site is indeed running lighttpd---but it appears to be
> an earlier version, because the official lighttpd site fails on this
> test.
>
> We have all the major web servers failing the test:
> * Apache 1.3
> * Apache 2.2
> * Microsoft-IIS/6.0
> * lighttpd/1.5.0
>
> So I think it's the python side that is wrong, regardless of what the
> standard is.
>
> What should I do next?
>
> I've rewritten the code to make it easier to test.  Just run it as is
> and it will try all my test cases; or pass in a site on the command
> line, and it will try just that.
>
> Thanks!
>
> #!/usr/bin/env python
> """Put the site you want to test as a command line parameter.
> Otherwise tests the list of defaults."""
>
> import urllib2
> import zlib
> import sys
>
> opener = urllib2.build_opener()
> opener.addheaders = [('Accept-encoding', 'deflate')]
>
> try:
>     sites = [sys.argv[1]]
> except IndexError:
>     sites = ['http://slashdot.org', 'http://www.hotmail.com',
>              'http://www.godaddy.com', 'http://www.linux.com',
>              'http://www.lighttpd.net', 'http://www.kenrockwel.com']
>
> for site in sites:
>     print "Trying: ", site
>     stream = opener.open(site)
>     data = stream.read()
>     encoded = stream.headers.get('Content-Encoding')
>     server = stream.headers.get('Server')
>
>     print "  %s - %s (%s)" % (site, server, encoded)
>
>     if encoded == 'deflate':
>         before = len(data)
>         try:
>             data = zlib.decompress(data)
>             after = len(data)
>             print "  Able to decompress...went from %i to %i." %
> (before, after)
>         except zlib.error:
>             print "  Errored out on this site."
>     else:
>         print "  Data is not deflated."
>     print