How to submit producer ready() changes, and tuning async core

[Donovan Baarda]

---------------------- multipart/mixed attachment
G'day Sam and medusa@e...,

sorry for appearing on this list outa nowhere announcing code changes
without serving the obligitory lurk time. Was just going to send this to Sam
then noticed there was a list.

I've been fiddling with medusa (ps, thanks for making it "Free!", this was
the final straw in convincing me it was worth using). And have made a few
changes you might like to fold back in. I was wondering how to go about
submitting changes etc. Have you got anything setup like a sourceforge
project for medusa?

I've added the ability for producers to block via a ready() method,
effectively allowing async producers. The changes required were suprisingly
small, affecting only asynchat.py and producers.py, plus some minor fixups
to http_server.py and filesys.py to avoid conflicts with the new feature. I
was using 20000601 as my base. Medusa appears to be working fine, and the
combined changes to asynchat.py and producers.py make them actualy simpler
and smaller than they were before. The changes are backwards compatible with
old producers that don't have a ready() method. The only conflicts in
http_server and filesys were pre-existing (and broken) ready() methods or
conflicts with my changes the composite_producer.

I've noticed that the zope guys early on mentioned that they were thinking
of extending medusa so that producers could block, probably by giving them a
ready() method. I've also seen hints of this throughout your code, with
vestigial ready() and stalled() methods, and even a writeable_for_proxy()
test in your http_server.

Dispite all this, no-one seems to have implemented it. I know that this
feature can be worked around by using dispatchers to push data onto an
asynchat, but the producer queue is a powerful feature, and it is nice
to be able to push asyc-producers onto that queue. I notice the ZServer guys
are even using the producer queue to push callbacks onto it using empty
producers that just execute the callback on more()... neat idea.

BTW, I saw someone else post on the egroups list about writeable_for_proxy
not working properly... AFAICT, it looks like it's something that was never
completely implemented. Using this ready() change will achieve all you
wanted and more :-) Warning! these changes are only lightly tested... 

I am planning to use medusa for on-demand mirroring with http, ftp, and
(eventualy) rsync back and front ends. The easy way to achieve this was to
make ftp and http filesystems that fetched and cached on demand. The problem
is all the filesystem stuff is blocking, so this wouldn't work well.
Implementing ready() for an async_file_producer looked easier than
implementing a non-blocking replacement for the filesystem layer.

On a different note, just read your stuff on recent improvements to the
async core. You mention the overhead of readable() and writeable() methods
in the polling loop. One trick that might be worth trying is something from
visibility tests in 3D graphics; cache the result. The idea is when you test
an object's visibility, give it an integer value. For each successive frame,
decrement the value, and don't test it's visibility again until the value
reaches zero. The size of integer can be tuned based on how soon you think
it's visibility status might change. The price is occasionaly you get it
wrong, but usually only for a few frames.

For async core stuff, you could 'learn' the visibility value based on
network load, whatever. If you made it robust enough to handle the
occasional "false write/readable" error, you could cache both 'writeable'
and 'not writable' type results. You would need to make sure it could handle
the occasional visibility error.

-- 
----------------------------------------------------------------------
ABO: finger abo@m... for more info, including pgp key
----------------------------------------------------------------------

---------------------- multipart/mixed attachment
A non-text attachment was scrubbed...
Name: medusa-ready.tar.bz2
Type: application/octet-stream
Size: 20173 bytes
Desc: not available
Url : http://mail.python.org/pipermail-21/medusa-dev/attachments/174af79d/medusa-ready.tar.bz2

---------------------- multipart/mixed attachment--