Slow network?

[Laszlo Nagy]

> It is very likely that nodelay is actually hurting you here.
>
> Using the select module and doing non-blocking IO will be faster than
> using threads for this as well.
>   
These sockets are non blocking and I'm using select.select indeed. Here 
is how it is implemented:

def read_data(self,size):
res = ""
fd = self.socket.fileno()
while not self.stop_requested.isSet():
remaining = size - len(res)
if remaining<=0:
break
# Give one second for an incoming connection so we can stop the
# server in seconds when needed
ready = select.select([fd], [], [], 0.2)
if fd in ready[0]:
data = self.socket.recv(min(remaining,8192)) # 8192 is recommended by 
socket.socket manual.
if not data:
# select returns the fd but there is no data to read -> connection closed!
raise TransportClosedError("Connection closed.")
else:
res += data
else:
pass
if self.stop_requested.isSet():
raise SystemExit(0)
return res

def write_data(self,data):
fd = self.socket.fileno()
while not self.stop_requested.isSet():
size = len(data)
if size==0:
break
# Give one second for an incoming connection so we can stop the server 
in seconds when needed
ready = select.select([], [fd], [], 0.2)
if fd in ready[1]:
sent = self.socket.send( # 8192 is recommended by socket.socket manual.
data[:max(size,8192)]
)
if not sent:
# select returns the fd but there is no data written -> connection closed!
raise TransportClosedError("Connection closed.")
else:
data = data[sent:] # Not too efficient...
else:
pass
if self.stop_requested.isSet():
raise SystemExit(0)

All other I/O methods are calling write_data() and read_data().

One thing I cannot do is to remove threads. This endpoint class is 
specifically designed so that other threads can put messages into the 
outgoing queue, and they do not need to wait for the message to be 
written out into the socket:

endpoint.send_message(msg) # won't block unless the outgoing queue is full
endpoint.recv_message(msg) # won't block unless the incoming queue is empty

Given this situation, I see no alternative to using threads. Do you?

Here is how the socket is set up on the client side:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,1)
s.connect((params['address'],params['port']))
return orb.endpoint.SocketEndpoint(s)


And on the server side:

def handle_request(self):
conn,client_address = self.serversocket.accept()
self.logger.info( str(client_address) + ' connecting')
conn.setsockopt(socket.IPPROTO_TCP,socket.TCP_NODELAY,1)
t = orb.endpoint.SocketEndpoint(conn,self.router)

The socket based endpoint starts with setting non-blocking mode:

class SocketEndpoint(Endpoint):
"""Specialized Transport endpoint that communicates over a socket."""
def __init__(self,socket,router=None):
self.socket = socket
self.socket.setblocking(0)
Endpoint.__init__(self,router)

Here are some more test results:

- original example with setblocking(0) and TCP_NODELAY: 130 messages / sec
- same example without TCP_NODELAY: 130 messages/sec (I don't understand 
why?)
- same example without setblocking(0): 130 messages/sec (I guess because 
I'm using select.select already?)
- same example without setblocking(0) and without TCP_NODELAY: 130 
messages / sec

Now I really didn't understand where is the problem? I suspected that 
maybe the overhead for select.select? But nope, tried the same example 
without select.select and without setblocking(0) and it was also 130 
messages/sec.

What is hurting me then? Is it not TCP? :-(