tornado.web ioloop add_timeout eats CPU

[Bryan]

Laszlo Nagy wrote:
[...]
> And here is my problem. If I point 5 browsers to the server, then I get
> 2% CPU load (Intel i5 2.8GHz on amd64 Linux). But why? Most of the time,
> the server should be sleeping. cProfile tells this:
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>          1    0.000    0.000  845.146  845.146 <string>:1(<module>)
>    1135775  832.283    0.001  832.283    0.001 {method 'poll' of
> 'select.epoll' objects}
>
> I have copied out the two relevant rows only. As you can see, total
> runtime was 845 seconds, and 832 seconds were spent in "epoll".
> Apparently, CPU load goes up linearly as I connect more clients. It
> means that 50 connected clients would do 20% CPU load. Which is
> ridiculous, because they don't do anything but wait for messages to be
> processed. Something terribly wrong, but I cannot figure out what?

What's wrong is the 1,135,775 calls to "method 'poll' of
'select.epoll' objects".
With five browsers waiting for messages over 845 seconds, that works
out to each  waiting browser inducing 269 epolls per second.

Almost equally important is what the problem is *not*. The problem is
*not* spending the vast majority of time in epoll; that's *good* news.
The problem is *not* that CPU load goes up linearly as we connect more
clients. This is an efficiency problem, not a scaling problem.

So what's the fix? I'm not a Tornado user; I don't have a patch.
Obviously Laszlo's polling strategy is not performing, and the
solution is to adopt the event-driven approach that epoll and Tornado
do well.

-Bryan