Twisted an several CPUs

[Mike Meyer]

Jp Calderone <exarkun at divmod.com> writes:

>>Has anyone experience high load and twisted?
>   Distributing load across multiple machines scales better than distributing it over multiple CPUs in a single machine.  If you have serious scalability requirements, SMP is a minor step in the wrong direction (unless you're talking about something like 128-way SMP on a supercomputer :)

I agree. In my experience, heavily loaded machines doing network stuff
tend to load on I/O, not CPU. So more machines works better than more
CPU - though more bandwidth to disk or the network may be equally
useful.

The exception is if they are doing very CPU-intensive things in
response to network requests. In the latter case, spinning those
CPU-intensive things off to a separate process is a good option for
distributing the load across multiple CPUs. And with Twisted, you've
already got the framework in place to do that. In this scenario,
Twisted becomes a big switch, passing work requests off to other
processes, and returning the results back. That should be plenty
quick.

There are, of course, exceptions to pretty much everything.

      <mike
-- 
Mike Meyer <mwm at mired.org>			http://www.mired.org/home/mwm/
Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.