[Python-Dev] Stackless Python

[Josiah Carlson]

> I'm working on a project (http://www.cakem.net/) that involves
> networking.  I have to do a lot of decoding of network data into an
> internal representation, and encoding internal representations into a
> network format.
> 
> The decoding part is going to present a problem, a problem that could
> easily be solved by continuations.
> 
> I want to right the code so that it will work when there is no guarantee
> that I'll have an entire message before I start decoding it.  This means
> the parser may have have to stop at any random point in time, and then
> be restarted again when more data is available.  I see no way to do this
> without continuations, or threads, and I refuse to use threads for
> something like this.
> 
> For now, I think I will do one of two things.  Either build in the
> assumption that I will have the entire messages before I start parsing,
> or organize parsing as a series of steps that can be aborted in the
> middle and then restarted, and use exceptions to track whether the step
> was fully completed or not.  Sort of a poor man's continuation.
> 
> Anyway, I'm not a Python developer, just a user.  I just wanted to add
> this to the things you consider when you decide what you intend to do
> next with Python.  Continuations are important, and AFAIK, the stackless
> mod is the only way to get them.

Eric,

I've taken a look at your protocol, and it looks to be simple enough to
just decode the header (maybe partially) and determine (based on the
length of the data received) if the entire message has been received yet.

Certainly it does end up reparsing the header a few times, but as long
as you only check to see if the message has been received on data read
(which is guaranteed to be at least 1 byte for select/poll read
events, and is commonly much larger), there shouldn't be too much
reparsing.

I've actually done the same thing with some nontrivial headers, and have
had no problems maxing out a 100mbit connection with a PII-400 and
a properly subclassed asyncore.dispatcher.  Even though it would be
computationally more efficient to use continuations or generators, they
are not necessary.  You will likely find more overhead with the
cryptographic portions of your protocol than you will reparsing the
header multiple times.

 - Josiah