fiber(cooperative multi-threading)

[Duncan Booth]

Akihiro KAYAMA <kayama at st.rim.or.jp> wrote:

> Thanks for your replies.
> 
> But current Python generator specification requires me:
> 
> def f1():
>     for x in foo(f2, "foo"): yield x
>     for x in foo(f2, "foo"): yield x
>     # XXX v = ... (I don't know how to do this)
>     for x in foo(f2, "v=%s world" % v, "OK"): yield x
> 
> I think it is not straitforward. Single level function which generator
> impose is impractical for real use.

Not just impractical, there are plenty of situations where you simply
cannot do that at all. 

Here's a greenlet example (from
http://socal-piggies.org/presentations/grig/2005_07_21/): 

from py.magic import greenlet
import xml.parsers.expat

def send(arg):
    greenlet.getcurrent().parent.switch(arg)

def start_element(name, attrs):
    send(('START', name, attrs))
def end_element(name):
    send(('END', name))
def char_data(data):
    data = data.strip()
    if data: send(('DATA', data))

def greenparse(xmldata):
    p = xml.parsers.expat.ParserCreate()
    p.StartElementHandler = start_element
    p.EndElementHandler = end_element
    p.CharacterDataHandler = char_data
    p.Parse(xmldata, 1)

def iterxml(xmldata):
    g = greenlet(greenparse)
    data = g.switch(xmldata)
    while data is not None:
        yield data
        data = g.switch()

if __name__ == "__main__":
    for data in iterxml("somexmldata"):
        # do something with data


The greenlet here calls expat, but the 'yield' happens inside an expat
callback. To get this to work you would need to rewrite expat to use its
callback as a generator. Expat is coded in C, so you also need to find
some way to get it to save the state of the C functions when it has
yielded. 

I think this example should answer Arnaud's question ("In fact it is not
clear to me at the moment what can be done (sensibly :) with the OP's
Fiber class that cannot be achieved with the run() function I
suggested.") Arnaud's run function cannot be used by anything which needs 
to save C stack state.

> I am happy if I could use convenient coroutine features via standard
> or simple extension library.  py.magic.greenlet may be what I'm
> looking for, but I wonder why this is named "magic" :-)

I think it is called "magic" because it does something which is at first 
glance impossible: saving the C stack state as well as the Python stack. 
Possibly it does something underhand to achieve an impressive effect.
I haven't looked at the implementation to see how it does it: creating 
threads and then scheduling them cooperatively would be one way but I 
expect it just switches the C stack around between different areas of 
memory.

A long time ago I ported the BCPL coroutine library and there is one 
function in the middle (the one which does the stack switch) that 
definitely felt like magic.

One interesting question raised by all this is whether those of us using 
Windows can usefully call Microsoft's Fiber functions through ctypes and 
get the benefit of coroutines without having to load any additional C 
extensions at all.