Python concurrent tasking frameworks?

[Michael Sparks]

Chris Stiles wrote:

> I was wondering if anyone had a list of the various (presuming more than
> one)
> Python tasking frameworks?  I know of Twisted already, but I'm really
> looking
> for something along the lines of a task/thread pool type of arrangement. 

The Kamaelia framework might or might not be what you're looking for. It's
not really thread pool based, but rather based around the idea of
communicating generators (which as used as components) to form
interesting systems. (Though you can also use threads if you want)

Communications happens in a manner very similar to unix pipelines, but
since it's a single system data is passed (essentially) by reference rather
than serialised to go over a unix pipe, and also rather than a single 1D
pipe, you can form arbitrary "graphlines".

A simple TCP based vorbis streaming client can look like this:

pipeline(TCPClient("127.0.0.1",clientServerTestPort),
         VorbisDecode(),
         AOAudioPlaybackAdaptor()
        ).run()

(Assuming necessary imports!)

This says take the data the TCPClient sends out and pass to the
VorbisDecode, and pass its output to the AOAudioPlaybackAdaptor.
The corresponding server accompanies this as example3 in the
distribution.

A simple (useful) example of a "graphline", looks like this:

Graphline(
     CHOOSER = Chooser(items = files),
     IMAGE = Image(size=(800,600), position=(8,48)),
     NEXT = Button(caption="Next", msg="NEXT", position=(72,8)),
     PREVIOUS = Button(caption="Previous", msg="PREV",position=(8,8)),
     FIRST = Button(caption="First", msg="FIRST",position=(256,8)),
     LAST = Button(caption="Last", msg="LAST",position=(320,8)),
     linkages = {
        ("NEXT","outbox") : ("CHOOSER","inbox"),
        ("PREVIOUS","outbox") : ("CHOOSER","inbox"),
        ("FIRST","outbox") : ("CHOOSER","inbox"),
        ("LAST","outbox") : ("CHOOSER","inbox"),
        ("CHOOSER","outbox") : ("IMAGE","inbox"),
     }
).run()

This forms a primitive slideshow - in which a "chooser" gets a list of
filenames, which depending on which button is pressed (next, previous,
first, last) sends filename to an Image display component for display.

(Again assuming a little bit of setup)

I gave a talk about Kamaelia at Europython, using Kamaelia, so if you feel
like grabbing a copy of the presentation which also includes everything you
need to get started, please look here:
   * http://tinyurl.com/cnk9c

That includes a more complex slideshow, though you need more than one
mouse button for it to be useful. (It can show graph/network topologies
dynamically, and can be more non-linear in terms of presentation)

There's a tutorial on creating components here:
   * http://tinyurl.com/dp8n7

A longer detailed introduction document here:
   * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

And the main website/docs here:
   * http://kamaelia.sourceforge.net/
   * http://kamaelia.sourceforge.net/Docs/

Depending on your focus Kamaelia contains tools for building TCP &
Multicast servers and clients relatively easily and is designed to naturally
encourage the use of concurrency easily and scalably. There's also a growing
collection of components for more aural/visual work with a simple Pygame
based "window manager" (really stretching things there, surface manager
might be more appropriate) for components, and audio decode/playback.

It isn't as mature as Twisted though, so that should affect your viewpoint!
(Had some good chats at Europython though about how we can go about
integration with twisted!)

<musing> I suppose it should be possible to have pools of existing
components for wiring into connections on demand though, which
might be interesting. </musing>

Best Regards,


Michael.
--
http://kamaelia.sourceforge.net/