Questions about asyncore

[Josiah Carlson]

On Jul 29, 4:09 am, Frank Millman <fr... at chagford.com> wrote:
> Hi all
>
> I have been using my own home-brewed client/server technique for a
> while, using socket and select. It seems to work ok. The server can
> handle multiple clients. It does this by creating a new thread for
> each connection. Each thread runs its own select loop.
>
> I am making some fairly big changes, so I thought I would look atasyncore. I modified my program to useasyncorewithout much trouble,
> and it feels nicer. It uses async I/O instead of threading, and it
> relieves me of having to run my own select loop.
>
> I have two questions. They both relate to whether I am using the
> module as intended. The documentation is rather sparse, and I know
> from experience that just getting something working is no guarantee
> that I am getting the full benefit.
>
> Firstly, having gotasyncoreworking, I had a look at asynchat. As far
> as I can see I get very little benefit from using it. I have already
> set up a 'messaging' protocol between server and client, where all
> messages consist of 5 digits for the message length, followed by the
> message. The message consists of a pickled tuple, where the first
> element is a message identifier, and the rest is the message body.
>
> This is how it works inasyncore-
>
>     def __init__(self,channel):
>         [...]
>         self.msgLength = 0
>         self.recvData = ''  # temporary buffer
>
>     def handle_read(self):
>         self.recvData += self.recv(8192)
>         if not self.msgLength:
>             if len(self.recvData) < 5:  # 1st 5 bytes = msg length
>                 return
>             self.msgLength = int(self.recvData[:5])
>             self.recvData = self.recvData[5:]
>         if len(self.recvData) < self.msgLength:
>             return
>         data = loads(self.recvData[:self.msgLength])
>         self.recvData = self.recvData[self.msgLength:]
>         self.msgLength = 0
>         [handle data]
>
> This is how I got it working in asynchat -
>
>     def __init__(self,channel):
>         [...]
>         self.recvData = ''  # temporary buffer
>         self.set_terminator(5)
>         self.gotMsgLength = False
>
>     def collect_incoming_data(self, data):
>         self.recvData += data
>
>     def found_terminator(self):
>         if self.gotMsgLength:  # what follows is the message
>             data = loads(self.recvData)
>             self.set_terminator(5)
>             self.gotMsgLength = False
>             [handle data]
>         else:  # what follows is the message length
>             self.set_terminator(int(self.recvData))
>             self.gotMsgLength = True
>         self.recvData = ''
>
> It may be slightly neater, but it does not seem worth adding an extra
> layer just for that. Does asynchat give me any other benefits I may
> have overlooked?

Giampaolo already offered up some information, but I will offer these
two little tidbits:
In your first client, your handle_read doesn't handle the case where
you have received multiple packets at one time.  That is, imagine that
in your one .read(8192), you received exactly two messages (the prefix
length and the pickle itself times two).  The first pass will pick up
the message and handle the data.  But unless the socket becomes
readable again, the second message will never be processed.  And even
if the socket becomes readable immediately in the next select() call,
the message will be delayed depending on what other sockets are up
to.  The asynchat module handles that case just fine.

As for push() vs. send(); send() returns the number of bytes sent.  If
it sends less than the total block of data (which can be the case with
large blocks of data, small TCP/IP windows over a slow connection, or
small TCP/IP buffers), you need to be aware of it and attempt to
resend the remaining.  Again, the asynchat module handles that case
just fine with it's .push() method; when it returns, you know that the
data to be transferred will be transferred as long as the connection
stays alive.  Without .push(), you need to implement that behavior
yourself (less boilerplate for every subclass = easier maintenance).

(I normally don't hop on this list to comment, so please cc me on any
replies)
 - Josiah

> My second question relates to writing a dummy client program to test
> the server. I just want to send it some messages and print the
> responses. Some messages incorporate data extracted from previous
> responses, so I have to wait for the reply before continuing.
>
> I am usingasyncorefor this as well. However, the only way I can
> think of to get it working is to runasyncore.dispatcher in a separate
> thread.
>
> To send messages, I append them to a list of messages to be sent. The
> dispatcher method writable() returns True if there is anything in the
> list, else False.
>
> To receive messages, I run a 'while 1' loop in the main thread, with a
> sleep of 0.1, until recvData has something in it.
>
> It works, but it seems odd to use a separate thread, as one of th
> points ofasyncoreis to avoid multi-threading. Is there a better way
> to write the client program?
>
> Thanks
>
> Frank Millman