fiber(cooperative multi-threading)

[Michael Sparks]

Duncan Booth wrote:
[ snip sections about why the single layer aspect can be helpful ]

> I'm happy with generators as they are: they're a great solution to a
> problem that most of us didn't realise we had until the solution came
> along. That doesn't mean that I wouldn't also like to have a separate
> coroutine library, but I would use it to replace situations that might
> otherwise have to use threads, not to replace generators.

Oh agreed. For that, it would probably be nice to see greenlets (or
something similar) make it into the standard library, but I suspect
that's too much of a jump at the moment.

I think for me, what we tend to do is prototype something as a stub
program, and either turn that into a threaded component or a generator
component. If I was to do something halfway I'd probably look at greenlets
at the moment, though so far we've not found a use for that. (which is
relatively suprising)

Any time it's looked useful, generally it's actually meant "spin this
control out to another smaller component". That's generally then resulted
in that spun out component being reusable.

For example, probable border case for coroutines is in this:
http://kamaelia.svn.sourceforge.net/viewvc/kamaelia/branches/private_MPS_Scratch/Apps/Kamaelia-Grey/App/greylisting.py?revision=3684&view=markup

We do do something that implies full coroutine support would be useful -
specifically the calls:
             yield WaitComplete(self.getline(), tag="_getline1")
...
                     yield WaitComplete(self.getline(), tag="getline2")

To:
   31     def getline(self):
   32         control_message = ""
   33         while 1:
   34             while not self.anyReady():
   35                 self.pause();
   36                 yield 1
   37             while self.dataReady("control"):
   38                 control_message = self.recv("control")
   39                 if isinstance(control_message, socketShutdown):
   40                     self.client_connected = False
   41             if self.dataReady("inbox"):
   42                 self.logging_recv_connection()
   43                 return
   44             else:
   45                 if not self.client_connected :
   46                     self.breakConnection = True
   47                     return
   48             yield 1

(This is of course faking full co-routines by getting the caller to call
something else temporarily)

The thing is, if we split that out to an external component, there's
potential benefits.

*shrug* IMO, code should be written for clarity first, so simply being able
to have this as pure co-routines may be useful. However, being able to tag
the calls like this:
             yield WaitComplete(self.getline(), tag="_getline1")
...
                     yield WaitComplete(self.getline(), tag="getline2")

Meant that I could run an introspector:
  506 Pipeline(
  507     PeriodicWakeup(),
  508     WakeableIntrospector(),
  509 ).activate()

And was able to see that some clients were misbehaving at different time
points, but also seeing clearly able to see what state the component was
waiting/hanging in when dealing with broken clients, and able to add in
timeouts trivally. cf:

*debug* 'THREADS'['Axon.Microprocess.microprocess_7068_getline1', 
'Kamaelia.Chassis.Graphline.Graphline_7062',
'Kamaelia.Chassis.Pipeline.Pipeline_7',
... snip ...
'Kamaelia.Internet.ConnectedSocketAdapter.ConnectedSocketAdapter_7059', 
'Kamaelia.Internet.Selector.Selector_11',
'Kamaelia.Internet.TimeOutCSA.ActivityMonitor_7060',
'Kamaelia.Internet.TimeOutCSA.ResettableSender_7061',
'__main__.GreylistServer_8',
'__main__.PeriodicWakeup_5',
'__main__.TCPS_10',
'__main__.WakeableIntrospector_6']

Now you could of course design a coroutine system for python to do that
and I'd hope if anyone did, they'd think of it being useful to trace the
where a co-routine is currently stuck :-)

All that said, my personal primary aim for kamaelia is to try and make it
into a general toolkit for making concurrency easy & natural (as well
as efficient) to work with. If full blown coroutines turn out to be part
of that c'est le vie :-)

I think it's a bit like the arguments around lambda being limited to a
single expresson vs a throwaway def. Both are useful, being stuck with one
or the other is more limiting than is perhaps ideal.

> Thanks you. I really do intend most of my posts to be helpful.

Likewise :-)

Have fun :-)


Michael.