[Python-Dev] RE: NT select.select?

[Guido van Rossum]

> It turns out that NT has a default 64 fd limit on arguments to
> select(). The good news is that you can actually bump the limit up 
> to whatever number you want by specifying a define when compiling 
> python15.dll.
> 
> If you have the ability to rebuild your python15.dll, you can add
> the define:
> 
> FD_SETSIZE=1024
> 
> to the preprocessor options for the python15 project to raise the
> limit to 1024 fds.
> 
> The default 64 fd limit is too low for anyone trying to run
> an async server that handles even a modest load, so I've 
> submitted a bug report to python.org asking that the define
> above find its way into the next python release...

Brian,

(Also in response to your bug report.)  I'm a little worried that
upping the limit to 1024 would cause some performance problems if
you're making a lot of select() calls.  The select allocates three
arrays of length FD_SETSIZE+3; each array item is 12 bytes.  This is a
total allocation of more than 36K for a meager select() call!  And all 
that memory also has to be cleared by the FD_ZERO() call.

If you actually have that many sockets, that's worth paying for (the
socket objects themselves use up just as much memory, and your Python
data structures for the sockets, no matter how small, are probably
several times bigger), but for a more typical program, I see this as a 
lot of overhead.

Is there a way that this can be done more dynamically, e.g. by making
the set size as big as needed on windows but no bigger?

(Before you suggest allocating that memory statically, remember it's
possible to call select from multiple threads.  Allocating 36K of
thread-local space for each thread also doesn't sound too pleasant.)

--Guido van Rossum (home page: http://www.python.org/~guido/)