High volume websites using Python web server software?

[Josiah Carlson]

Irmen de Jong <irmen at -nospam-remove-this-xs4all.nl> wrote:
> 
> Ian Bicking wrote:
> > I think he's talking about servers that server a higher-than-normal 
> > number of simultaneous connections.
> 
> Well, not necessarily so, but this is an even better description.
> I was looking for web sites with moderate visitor counts
> (where 'high' means Yahoo, Google, Cnn, and 'low' means a
> 200-user forum about automobiles) that are running on one of Python's
> web application servers / network servers.
> 
> If it's a site that uses an async I/O framework, and it actually
> does sustains the potential very large amount of simultaneous
> connections, even better.
> Actually, it doesn't even have to be a web site. A 10.000-user IRC
> server in Python would also qualify :)
> 
> Why am I asking this?
> Someone on a programming forum elsewhere was sceptical that you
> can create high performance Python network servers.
> I suggested Twisted or Medusa. He asked for example servers.
> So there you are :)
> 
> 
> Anyway I've got a few pointers from the replies already given.
> Thanks!

Using a heavily customized variant of asyncore, I have written a client
that can handle 500 simultaneous connections (default file handle limit
with Python is 512) on a modern (1+ghz) processor.  It is actually
limited by bandwidth and latency on our 100mbit connection, as the
protocol is fairly chatty.

We have benchmarked it (each command results in a delay of
.25/random.random() seconds, timeouts set at 30 seconds), and it comes
out to roughly 4.8 million of these connections per day on a single
processor (we are unsure if it scales running multiple processes on a
multiple processor machine, as 99% of the processor use is system-level
select.select() calls).

We are actually concerned with accidentally DOSing remote servers that
have less efficient server implementations.

Is that high enough volume?


 - Josiah