massive threading performance (was:Re: UDP performance)
Paul Sijben
sijben at eemvalley.com
Thu Apr 20 06:05:47 EDT 2006
OK the problem I posted about earlier is NOT a UDP/socket problem, it is
a threading problem. Albeit one that only happens when you have many
thrreads????
I have made two little scripts using the code I copied below, one client
and one server (attached).
If I run them concurrently on the same machine I see the following times:
>>1145526192.825508
<<1145526192.825848
>>1145526193.829325
<<1145526193.834927
a transfer time in the milliseconds. Much less than the times I see in
the full application.
OK so I put them both in a multithreaded script (also attached)
>>1145526971.619558
<<1145526971.619909
>>1145526972.619241
<<1145526972.619647
again transfer time in milliseconds.
Not like this that I get from the profile of my code:
<< 1145517554.363850 send
>> 1145517554.647485 recv
which uses the same communication and threading but has 20+ threads?
Now I am completely baffled!
again I really appreciate if anyone can shed light on this!
Paul
Paul Sijben wrote:
> I am stumped by the following problem. I have a large multi-threaded
> server accepting communications on one UDP port (chosen for its supposed
> speed).
>
> I have been profiling the code and found that the UDP communication is
> my biggest drain on performance! Communication where the client and the
> server are on the same machine still takes 300ms or sometimes much more
> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>
> I must be doing something wrong and would really appreciate feedback on
> my code below:
>
> I open the server port with
>
> self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> self.s.bind((myaddress, myport))
>
> I then open a client port with
>
> self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> self.s.connect((host, port))
>
> the client sends data with
>
> self.s.sendall(data)
>
> and the server with
>
> self.s.sendto(data,link.remoteaddress)
>
> both receive with
>
> buf, address = socket.recvfrom(8192)
>
> The sender and receiver are in separate threads (threading.Thread).
>
> Does anyone know what is going wrong here, the socket communication, the
> thread scheduling?
>
> Paul Sijben
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: client.py
URL: <http://mail.python.org/pipermail/python-list/attachments/20060420/4a9afb73/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: server.py
URL: <http://mail.python.org/pipermail/python-list/attachments/20060420/4a9afb73/attachment-0001.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mthreadpy.py
URL: <http://mail.python.org/pipermail/python-list/attachments/20060420/4a9afb73/attachment-0002.ksh>
More information about the Python-list
mailing list