massive threading performance (was:Re: UDP performance)

[Paul Sijben]

OK the problem I posted about earlier is NOT a UDP/socket problem, it is
a threading problem. Albeit one that only happens when you have many
thrreads????

I have made two little scripts using the code I copied below, one client
and one server (attached).

If I run them concurrently on the same machine I see the following times:
>>1145526192.825508
<<1145526192.825848

>>1145526193.829325
<<1145526193.834927

a transfer time in the milliseconds. Much less than the times I see in
the full application.

OK so I put them both in a multithreaded script (also attached)


>>1145526971.619558
<<1145526971.619909

>>1145526972.619241
<<1145526972.619647

again transfer time in milliseconds.

Not like this that I get from the profile of my code:
<< 1145517554.363850 send
>> 1145517554.647485 recv

which uses the same communication and threading but has 20+ threads?

Now I am completely baffled!

again I really appreciate if anyone can shed light on this!

Paul


Paul Sijben wrote:
> I am stumped by the following problem. I have a large multi-threaded
> server accepting communications on one UDP port (chosen for its supposed
> speed).
> 
> I have been profiling the code and found that the UDP communication is
> my biggest drain on performance! Communication where the client and the
> server are on the same machine still takes 300ms or sometimes much more
> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
> 
> I must be doing something wrong and would really appreciate feedback on
> my code below:
> 
> I open the server port with
> 
> 	self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> 	self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> 	self.s.bind((myaddress, myport))
> 
> I then open a client port with
> 
> 	self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> 	self.s.connect((host, port))
> 
> the client sends data with
> 
> 	self.s.sendall(data)
> 
> and the server with
> 
> 	self.s.sendto(data,link.remoteaddress)
> 
> both receive with
> 
> buf, address = socket.recvfrom(8192)
> 
> The sender and receiver are in separate threads (threading.Thread).
> 
> Does anyone know what is going wrong here, the socket communication, the
> thread scheduling?
> 
> Paul Sijben
> 


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: client.py
URL: <http://mail.python.org/pipermail/python-list/attachments/20060420/6a36b4a2/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: server.py
URL: <http://mail.python.org/pipermail/python-list/attachments/20060420/6a36b4a2/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mthreadpy.py
URL: <http://mail.python.org/pipermail/python-list/attachments/20060420/6a36b4a2/attachment-0002.ksh>