[issue16873] increase epoll.poll() maxevents default value, and improve documentation

[Charles-François Natali]

Charles-François Natali added the comment:

The implementation can't simply drain a queue, because it can be
level-triggered (which is the case by default), so you want to keep
events around. The kernel actually uses a red-black tree, but I didn't
really check how it's done (there's probably very good reasosns for
that).

Anyway, it can turn out to be a problem, for two reasons:
- performance: by specifying a maxevents value too low, several calls
to epoll_wait() must be made, instead of being able to process all
events at once
- the main problem that came to my mind is really starvation: let's
say you have 2*FD_SETSIZE client sockets registered in your poll
object. The first call to epoll_wait() returns sockets from 0 to
FD_SETSIZE-1: you process them, so they're not ready anymore. The next
call returns the clients from FD_SETSIZE to 2*FD_SETSIZE, same thing.
But by the time you call epoll_wait() for the third time, if the first
FD_SETSIZE clients are ready again, they will be returned, etc. So the
2*FD_SETSIZE th client may very well never be reported ready: that's
starvation.

I actually wrote a script to reproduce this issue:
"""
$ ./python /home/cf/test_epoll.py
Working with 4080 FDs, -1 maxevents
Number of missing FDs:4080
Number of ready FDs: 1023
Number of missing FDs:3057
Number of ready FDs: 0
Number of missing FDs:3057
Number of ready FDs: 1023
Number of missing FDs:2034
Number of ready FDs: 0
Number of missing FDs:2034
Number of ready FDs: 1023
Number of missing FDs:2034
Number of ready FDs: 0
Number of missing FDs:2034
Number of ready FDs: 1023
Number of missing FDs:2034
Number of ready FDs: 0
[...]
"""

If you specify a large enough maxevents:
"""
$ ./python /home/cf/test_epoll.py 64000
Working with 4080 FDs, 64000 maxevents
Number of missing FDs:4080
Number of ready FDs: 4080
"""

Note that it's really a corner issue, but I stumpled upon this problem
while writing a test in issue #16853, and several projects (Tulip,
Tornado, pyftpdlib) fall into this trap.

I see several options:
1) just keep it that way (i.e. with maxevents set to FD_SETSIZE), and
add a note in the documentation. I think it's always better to handle
problems in the library than let the users get bitten.
2) increase `maxevents` default value. The problem is that I don't
like magic values, and a too large value could incur increased memory
consumption (and with the current implementation reduced performance
because the epoll events buffer is allocated at each poll(), see issue
#16876.
3) use a simple heuristic: start with a reasonable value for
`maxevents` (FD_SETSIZE seems like a good candidate), and if
epoll_wait() ever returns `maxevents` events, double the value (that's
what libevent does, with a capping to 32000).

----------
Added file: http://bugs.python.org/file28591/test_epoll.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue16873>
_______________________________________