[Python-Dev] RE: NT select.select?

[Brian Lloyd]

Guido wrote:
>
> Brian,
> 
> (Also in response to your bug report.)  I'm a little worried that
> upping the limit to 1024 would cause some performance problems if
> you're making a lot of select() calls.  The select allocates three
> arrays of length FD_SETSIZE+3; each array item is 12 bytes.  This is a
> total allocation of more than 36K for a meager select() call! 
>  And all 
> that memory also has to be cleared by the FD_ZERO() call.
> 
> If you actually have that many sockets, that's worth paying for (the
> socket objects themselves use up just as much memory, and your Python
> data structures for the sockets, no matter how small, are probably
> several times bigger), but for a more typical program, I see 
> this as a 
> lot of overhead.
> 
> Is there a way that this can be done more dynamically, e.g. by making
> the set size as big as needed on windows but no bigger?
> 
> (Before you suggest allocating that memory statically, remember it's
> possible to call select from multiple threads.  Allocating 36K of
> thread-local space for each thread also doesn't sound too pleasant.)
> 
> --Guido van Rossum (home page: http://www.python.org/~guido/)

Hmm - after going through all of the Win32 sdks, it doesn't
appear to be possible to do it any other way than as a -D 
option at compile time, so optimizing for the common case
(folks who _don't_ need large numbers of fds) is reasonable.

Since we distribute a python15.dll with Zope on windows, this
isn't that big a deal for us - we can just compile in a higher
limit in our distributed dll. I was mostly thinking of the win32
users who don't have the ability to rebuild their dll, but
maybe this isn't that much of a problem; I suspect that the 
people who write significant socket apps that would run into 
this problem probably have access to a compiler if they need it.


Brian Lloyd        brian@digicool.com
Software Engineer  540.371.6909              
Digital Creations  http://www.digicool.com