Async Client with 1K connections?

[Dave Brueck]

> Before I take the plunge, I'd appreciate any advice on the feasibility
> and degree of difficulty of the following...
>
> I need extremely efficient and robust _client_ software for some
> common protocols like HTTP and POP3, supporting 1,000 simultaneous
> independent connections and commensurate network throughput.  The data
> get written to files or sockets, so no GUI needed.

1000+ connections is not a problem, although (on Linux at least, and probably
others) you'll probably need to make sure your process is allowed to have open
more file descriptors, especially if you're turning around and writing data to
disk (since that uses file descriptors too). This is OS-specific and has
nothing to do with Python, but IIRC you can do something like
os.sysconf(os.sysconf_names['SC_OPEN_MAX']) to see how many fd's your process
can have open.

> I am not a Python programmer :-( but I am a "fan" :-) and I have been
> reading about asyncore/Medusa/Twisted -- which would be my best bet?

You're probably going to be ok either way, but what are your throughput
requirements exactly? Are these connections pulling down HTML pages and small
images or are they big, multi-megabyte downloads? How big is your connection?
For 99% of uses asyncore or Twisted will be fine - but if you need very high
numbers of new connections per second (hundreds) or throughput (hundreds of
Mbps) then you might need to modify the framework or build your own - still in
Python but more tailored to your specific needs - in order to get those levels
of performance.

-Dave