Proposal for adding Shallow Threads and a Main Loop to Python

[Rhamphoryncus]

Diez B. Roggisch wrote:
> Hi,
>
> a few questions:
>
> > A shallow thread is just a generator modified in the most obvious
way
> > possible.  The yield statement is replaced with a waitfor
expression.
> > You give it the object you wish to "wait for".  Then when it's
ready
> > you get back a return value or an exception.  These waitfor
expressions
> > are the only points where your shallow thread may get suspended, so
> > it's always explicit.  If you call another function it will be
treated
> > exactly as a normal function call from a generator (this is why
they're
> > 'shallow'), so there's no issues with calling C functions.
> >
> > On the other end of things, your shallow thread object would have
> > __resume__ and __resumeexc__ methods (or perhaps a single
__resume__
> > method with a default raise_=False argument).  They return a tuple
of
> > (mode,value) where mode is a string of 'waitfor', 'exception', or
> > 'return' and value corresponds to that.  These methods will be used
by
> > a main loop to resume the shallow thread, like next() is used with
a
> > generator.
>
> So basically a shallow thread could be written in today's python like
this?
>
> class ShallowThread(object):
>     def __init__(self, data=None):
>         self.data = data
>
>     def run(self, initial_data=None):
>         while True:
>             ....
>             yield <some_state>
>             new_data = self.data
>             ....
>
> st = ShallowThread().run(<some_initial_data>)
> while True:
>     result = st.next()
>     st.data = <some_subsequent_data>
>
> Basically <some_state> is your above mentioned tuple, and waitfor is
like
> yield with a return value that I modeled with explicit state called
data.
> Is that correct so far?

Yes, that's the general idea.  I would give the following example
however, using Twisted 2.0's deferredGenerator facility (and assuming
all the functions I depend on are still modified the same way, just for
a fair comparison).

from twisted.internet.defer import *
from twisted.internet import reactor

def get_and_save(path):
    thingy = waitForDeferred(urllib.urlopen(path, async=True))
    yield thingy
    infile = thingy.getResult()
    thingy = waitForDeferred(open(split('/')[-1], async=True)
    yield thingy
    outfile = thingy.getResult()
    thingy = waitForDeferred(infile.read(async=True))
    yield thingy
    data = thingy.getResult()
    thingy = waitForDeferred(outfile.write(data, async=True))
    yield thingy
    thingy.getResult()  # Still needed to raise exceptions
    infile.close()
    outfile.close()
get_and_save = deferredGenerator(get_and_save)

def main():
    a = get_and_save("http://python.org/pics/PyBanner021.gif")
    b = get_and_save("http://python.org/pics/pythonHi.gif")
    c = get_and_save("http://python.org/pics/PythonPoweredSmall.gif")

    thingy = waitForDeferred(allDone(a, b, c))
    yield thingy
    thingy.getResult()
main = deferredGenerator(main)

if __name__ == "__main__":
    d = main()
    d.addBoth(lambda _: reactor.stop())
    reactor.run()

>
> > import mainloop, urllib
> >
> > def get_and_save(path):
> >     infile = waitfor urllib.urlopen(path, async=True)
> >     outfile = waitfor open(path.split('/')[-1], async=True)
> >     waitfor outfile.write(waitfor infile.read(async=True),
async=True)
> >     infile.close()
> >     outfile.close()
> >
> > def main():
> >     a = get_and_save("http://python.org/pics/PyBanner021.gif")
> >     b = get_and_save("http://python.org/pics/pythonHi.gif")
> >     c =
get_and_save("http://python.org/pics/PythonPoweredSmall.gif")
> >
> >     waitfor allDone(a, b, c)
> >
> > if __name__ == "__main__":
> >     mainloop.runUntil(main())
> >
> > Well there you have it.  I've glossed over many details but they
can be
> > cleared up later.  What I need to know now is how everybody else
thinks
> > about it.  Is this something you would use?  Does it seem like the
> > right way to do it?  And of course the all important one, can I get
it
> > in to python core?  <0.5 wink>
>
> I've difficulties grasping where the actual work is done - the event
> notifier thingies are sort of generators themselves, and the mainloop
gets
> them and calls some execute method on them?

It depends.  For files, the mainloop would have something like a
select-based polling loop.  Once something is ready, it would set the
appropriate event notifier to the "done" state and notify everything
waiting on it.


> And now the final $1000,0000.00 question - why all this? No offense
intended
> - it's a little bit more comfortable than the generators approach
sketched
> by others (e.g. David Mertz if I recall corretly) - but to my view,
it
> _could_ be done in today python because we have generators. Or not?

Go and compare the two get_and_save functions.  Would you call that a
"little bit more comfortable"?  Now imagine the function was 20 lines
to begin with, involving a couple loops and some list comprehensions.
Then just for kicks, imagine it without generators, instead turning it
into a state machine.

Yes, you can *technically* do it using generators (and they're what
make the implementation easy).  But you could also do it using a state
machine.  Doesn't mean it's practical though.

--
Adam Olsen, aka Rhamphoryncus