Python Parallel Paradigm

[Michael]

Sandy wrote:
...
> Lots of trees, but where's the Wood?
> 
> Where are concurrency/distributed models compared and discussed?


I don't know about wood, but you can find a shrubbery[*] called Kamaelia
sitting here:

   * http://kamaelia.sourceforge.net/Home

A basic (albeit now aging) tutorial as to how to take some existing code and
create components which can be used is here:
   *
http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi?rm=viewpost&nodeid=1113495151

Perhaps usefully, there's an overview of Kamaelia in this whitepaper:
   * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

It does miss out work from the past 18 months work since it was written
though.

Also, every single example you see here:
   * http://kamaelia.sourceforge.net/Cookbook.html

Is rather heavily parallel, except we just call our units of execution
components, components then talk in a CSP style fashion. The problem
you've *REALLY* got is making things accessible to the average developer.

That's the target audience for Kamaelia. ( If it doesn't work for that kind
of person, it's a conceptual bug in the system. )

As a result we normally refer to components being really simple,
communicating with the outside world via inboxes and outboxes (think
a person at a desk with some in-trays and out-trays).

Why? Because this is directly analogous to the ideas in both CSP *and* Unix
pipelines - the latter has been long understood by unix people as a way of
making it easier to build systems, the former understood largely by people
interested in parallelism as a good mechanism.

It also provices a nice route for development of new components - you
simply solve the core of the problem you wanted to solve, and then
remove the inputs back to inboxes and remove outputs back to outboxes.
It doesn't work for every class of problem, but does for a suprising
number.

We're in the process of revamping the website at the moment (it's November).
But for the moment, a simple tutorial on how the core concurrency tools in
Kamaelia fundamentally work is here:
   * http://kamaelia.sourceforge.net/MiniAxon/

(The tutorial was originally written for a pre-university who'd learnt
python the previous week)

One interesting fact though - we have an application for collaborative
whiteboarding where each whiteboard is both a client & server, and they
can form a collaboration tree - whereby everything scribbled and said (each
whiteboard contains an audio mixer) is distributed to everyone connected in
a P2P like fashion. It's a classic example of a desktop application with a
small twist, and was written because we needed it. 

The reason it's interesting is because it's highly parallel, with around
70 active components normally. *IF* the baseclass was changed to put
the generators in seperate threads, and ran under a Python VM that could
actually make the threads utilise seperate CPU's effectively (as allegedly
the IronPython VM does), then you'd be able to utilise a 60, 70 odd
multicore CPU with no change to the application code.

It's a nice example of a real world application written to solve a specific
need (we split the team multisite and needed an audio & scribbling based)
collaboration tool which is able to work effectively on existing hardware
systems, and *SHOULD* be trivially scalable to massively multicore systems
as VMs and CPUs become available, with no real change to the application
code. (Small change to Axon - our concurrency mechanism would be needed as
noted above. Obviously at the moment that change makes no sense to us)

It's also interesting because it was simpler to write due to the way we
make concurrency available (by thinking in terms of components that form
pipelines & graphlines), rather than despite using concurrency. (The initial
version took around 3 days)

If you want more detail on the whiteboard, there's been a tutorial in the
past month's Linux Format (ie December issue).

If you're interested in discussing anything concurrency related, and need a
forum to do so, you're more than welcome to use the Kamaelia mailing list
for doing so.

Kamaelia's hardest core goal is to make concurrency actually USEFUL for the
average developer. (though we tend to target maintenance over initial
development - which doesn't always makes life easier initially) That said,
we seem to be getting somewhere.

> There's Stackless Python (which I can't make head or tail of; I have been
> unable to find any lucid overview,

Finally, if the descriptions you find on the Kamaelia website don't make
sense to you, PLEASE tell us. I consider that as much of a bug as anything
else. (and we can't know that without being told)

Regards,


Michael.
--
Kamaelia Project Lead/Dust Puppy
http://kamaelia.sourceforge.net/Home
http://yeoldeclue.com/blog

[*] (yes, it's a deliberate mispelling of Camellia, which is a shrubbery,
     rather than a backronym)