"HTTP error -1" from urllib2

[John Nagle]

John J. Lee wrote:
> John Nagle <nagle at animats.com> writes:

> Can you create an example (preferably small) that fails?  Feel free to
> email it to me if it includes something you don't want to post.

    It's not a Python problem, as it turns out.  It's a problem in,
surprisingly, Coyote Point load balancers.

This fails:
====
telnet www.coyotepoint.com 80
GET / HTTP/1.0
Host: www.fourmilab.ch
User-agent: am

====

This works:
====
telnet www.coyotepoint.com 80
GET / HTTP/1.0
Host: www.fourmilab.ch
User-agent: an

=====

Note the difference in the "User-agent" field; "m" vs. "n".

     There's some problem in Coyote Point Equalizer load balancers
in USER-AGENT parsing.  If it sees a USER-AGENT string ending in
"m" but with no earlier "m" in the string, and the USER-AGENT field
is the last field in the HTTP header, it drops the packet.  One can make
this happen talking to the HTTP server with a Telnet client.
If you paste the sections between "===" lines above into a Windows
command line window, you can demonstrate this too.  (Remember to
copy the blank line that ends the header.)

     We found this because we were using a user agent string of
"SiteTruth.com rating system", which ends in "m" but doesn't
contain any other "m" characters.  A site run by people we know
wouldn't respond, and we've been working to figure out why.  They
own a Coyote Point Equalizer, and after much digging through
log files, it became clear that the load balancer was dropping
these packets, even though it wasn't configured to do so.
So we tried Coyote Point's own site, and it has exactly the
same problem.  It's thus probably a generic problem with
Coyote Point load balancers.  It's not a configuration
problem; we've checked the load balancer's configuration file.

     That load balancer uses regular expressions to parse HTTP
headers.  My guess is that we're going to find a "\m" somewhere
that a "\n" was intended.

     I'll be on the phone to Coyote Point on Monday.

				John Nagle