ValueError: filedescriptor out of range in select()

[Jean-Paul Calderone]

On Tue, 17 Mar 2009 18:03:59 +0100, Laszlo Nagy <gandalf at shopzeus.com> wrote:
>
>>For whatever reason, you're ending up with a lot of open files and/or 
>>sockets
>>(and/or any other resource based on file descriptors).  That results in new
>>file descriptors having large values (>=1024).
>>
>>You cannot use select() with such file descriptors.  Try poll() instead,
>>or Twisted. ;)
>
>Poll is not good, because it does not wait.

This is not true.  You can accomplish just what you're doing with select()
using poll() instead.  Oops, but I forgot, Windows doesn't have poll(), so
that doesn't really help you.  There's WaitForSingleObject if you can depend
on pywin32, though.  It supports timeouts, just as select() does (in fact,
select() is a wrapper around WaitForMultipleObjects on Windows - the multi-
handle version of WaitForSingleObject).

> [snip]
>
>You are talking about twisted - can someone tell me how twisted does this?

Twisted does it by taking the call to low-level I/O routines out of your
hands and just doing them right on its own. ;)  It also doesn't use blocking
I/O calls, and it doesn't make any I/O calls unless it thinks they're going
to succeed immediately anyway.

It deals with the problem of allowing timely exits on Windows by waiting
up select frequently, which is just a hack to work around the bug I
mentioned in CPython on Windows.

> [snip]
>
>I would really like to know what other options I have to implement this 
>efficiently. Is my approach bad?

I prefer the approach Twisted (and other libraries too) take.  Don't use
threads to multiplex I/O.  Use non-blocking I/O calls and select() or the
best platform-specific equivalent.

>
>Or am I using the wrong language? I started to use threads heavily in the 
>past months. Haskell might be a better choice for me, except that Python is 
>much more elegant. :-)

I'm sure people will tell you that this is a good use of threads and that
Python is up to the task.  I think it's a bad use of threads, but with a
different approach, I still think Python is up to the task.

Jean-Paul