[ python-Bugs-852837 ] FD_SETSIZE not having effect (old request 210843)

[SourceForge.net]

Bugs item #852837, was opened at 2003-12-02 20:16
Message generated for change (Comment added) made by miyamoto42
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=852837&group_id=5470

Category: Windows
Group: Python 2.3
Status: Open
Resolution: None
Priority: 5
Submitted By: Krister (miyamoto42)
Assigned to: Thomas Heller (theller)
Summary: FD_SETSIZE not having effect (old request 210843)

Initial Comment:
I think the FD_SETSIZE introduced by the fix for request 
210843 isn't working properly. 

According to some information on socket.com 
(http://www.sockets.com/winsock.htm#Deviation_MaxSo
ckets) you must define FD_SETSIZE before invoking 
winsock.h. 

Major guesswork follows:

This only define I can find is currently in selectmodule.c, 
but errnomodule.c is invoking winsock.h before if I'm 
reading the code right (unlikely) and the result is a 
limitation of 64 connections per thread when using select
() instead of the intended 512. This is causing 
considerable amount of problem with the BitTorrent 
clients under Windows (In the BitTorrent group referred 
to the "CPU hosing" problem). 

I'm constantly getting the error message "10055, An 
operation on a socket could not be performed because 
the system lacked sufficient buffer space or because a 
queue was full" when BitTorrent reaches exactly 64 
connections. 

Python 2.3.2
Window XP Home SP1 (Build 2600.xpsp2.030422-1633: 
Service Pack 1)
BitTorrent client 3.3 (http://bitconjurer.org/BitTorrent/)

/Krister

----------------------------------------------------------------------

>Comment By: Krister (miyamoto42)
Date: 2003-12-04 20:36

Message:
Logged In: YES 
user_id=871016

Ok, thanks for the response. 

When I try the example from theller I get this error message:

select.error: (10055, 'An operation on a socket could not be 
performed because the system lacked sufficient buffer space 
or because a queue was full')

I get this error message every attempt I do with value larger 
than 64.

Since your error message is what I should have got if I had 
been right and my error obviously originates in Windows, I can 
only agree that it isn't Python that is causing my problems, 
but as you say, probably either my network driver or my 
version of Windows.

Sic. I suppose I will have to find someone else to blame for 
my problems. :-)

----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2003-12-04 06:44

Message:
Logged In: YES 
user_id=31435

The FD_SETSIZE fiddling works fine -- lots of projects rely on 
it.  Not all flavors of Windows or socket providers support 512 
sockets, though.  FD_SETSIZE is simply a number larger than 
which Microsoft guarantees will not work in a select() call; 
they don't guarantee that a smaller number of sockets *will* 
work.  For example, 98SE starts to fail after about 50 sockets 
have been opened (in C or in Python) regardless of how large 
FD_SETSIZE is set, and regardless of how many distinct 
processes are involved in opening those sockets.

Note that winsock.h isn't an executable file -- it's just an 
include file, and so is never "invoked".  errnomodule.c has no 
reason to care how big *its* copy of winsock.h thinks 
FD_SETSIZE is because errnomodule.c makes no use of any 
fd_set.  selectmodule.c is the only piece of Python that can 
create an fd_set with more than 1 entry, and FD_SETSIZE is 
forced to 512 before its copy of winsock.h is included.

----------------------------------------------------------------------

Comment By: Thomas Heller (theller)
Date: 2003-12-03 20:39

Message:
Logged In: YES 
user_id=11105

I don't think this is Python bug, and it seems 512 sockets
are allowed in a select call.  This code:

from socket import *
from select import select
s = [socket(AF_INET, SOCK_DGRAM) for i in range(512)]
select(s, [], [], 1)

works fine (it waits one second), and, if I replace the 512
with 513 raises an exception:

Traceback (most recent call last):
..File "<stdin>", line 1, in ?
ValueError: too many file descriptors in select()

Can it be that this is a limitation of your network driver?
I've seen very weird behaviours in buggy drivers...


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=852837&group_id=5470