Proposal for adding Shallow Threads and a Main Loop to Python

[Rhamphoryncus]

First a bit about myself.  I've been programming in python several
years now, and I've got several more years before that with C.  I've
got a lot of interest in the more theoretical stuff (language design,
component architectures, etc).  Of late my focus has been on concurrent
operations (and on how to design a GUI architecture, but that's not
what this post is about).  I've looked at threads, and the inability to
kill them easily was a problem.  To get them to quit at all you had to
explicitly check a var that'd trigger it.  Might as well be using
event-driven.. so I looked at event-driven.  That doesn't have the
issue with ending operations, but it does make writing code much
harder.  Anything more than the most trivial loop has to be turned into
a state machine instead.

But recently I came up with a solution to that too.  I call it Shallow
Threads.

A shallow thread is just a generator modified in the most obvious way
possible.  The yield statement is replaced with a waitfor expression.
You give it the object you wish to "wait for".  Then when it's ready
you get back a return value or an exception.  These waitfor expressions
are the only points where your shallow thread may get suspended, so
it's always explicit.  If you call another function it will be treated
exactly as a normal function call from a generator (this is why they're
'shallow'), so there's no issues with calling C functions.

On the other end of things, your shallow thread object would have
__resume__ and __resumeexc__ methods (or perhaps a single __resume__
method with a default raise_=False argument).  They return a tuple of
(mode,value) where mode is a string of 'waitfor', 'exception', or
'return' and value corresponds to that.  These methods will be used by
a main loop to resume the shallow thread, like next() is used with a
generator.

A main loop is the next part of my proposal.  Where shallow threads
could be added with a fairly small patch, a main loop would require a
much more extensive one.  And unfortunately you need a main loop to use
shallow threads.  (You could use twisted but that's got the problems of
being third-party and not designed for inclusion in the python core.)

First part is the concept of an "event notifier object".  This is
simply an object that is not "done" yet, and you can "watch" and be
given a single value (or exception) when it is done.  This could be
something internal to your program like a shallow thread above, or it
could be something external like a file read completing.  This would,
like most other protocols in python, involve setting an attribute or a
method to support it.  I haven't yet figured out the best design
though.

We need versions of many existing functions that produce event
notifiers instead.  I suggest adding an async keyword to the existing
functions, defaulting to False, to indicate an event notifier should be
produced.  For example: waitfor (waitfor open("hello.txt",
async=True)).read(async=True).  At some mythological point in the
future, perhaps the default for async to be switched to True and that
example would get much shorter.

Now as for the main loop itself, I suggest a mainloop module.  The
primary function used would be mainloop.runUntil(), which would take an
event notifier as it's argument and would return as soon as that event
notifier was triggered.  An example of this and other features follows.

import mainloop, urllib

def get_and_save(path):
    infile = waitfor urllib.urlopen(path, async=True)
    outfile = waitfor open(path.split('/')[-1], async=True)
    waitfor outfile.write(waitfor infile.read(async=True), async=True)
    infile.close()
    outfile.close()

def main():
    a = get_and_save("http://python.org/pics/PyBanner021.gif")
    b = get_and_save("http://python.org/pics/pythonHi.gif")
    c = get_and_save("http://python.org/pics/PythonPoweredSmall.gif")

    waitfor allDone(a, b, c)

if __name__ == "__main__":
    mainloop.runUntil(main())

Well there you have it.  I've glossed over many details but they can be
cleared up later.  What I need to know now is how everybody else thinks
about it.  Is this something you would use?  Does it seem like the
right way to do it?  And of course the all important one, can I get it
in to python core?  <0.5 wink>

--
Adam Olsen, aka Rhamphoryncus