[Python-Dev] Pythonic concurrency

[Michael Sparks]

Hi Bruce,


On Thursday 06 October 2005 18:12, Bruce Eckel wrote:
> Although I hope our conversation isn't done, as he suggests!
...
> At some point when more ideas have been thrown about (and TIJ4 is
> done) I hope to summarize what we've talked about in an article.

I don't know if you saw my previous post[1] to python-dev on this topic, but 
Kamaelia is specifically aimed at making concurrency simple and easy to use. 
Initially we were focussed on using scheduled generators for co-operative 
CSP-style (but with buffers) concurrency.
   [1] http://tinyurl.com/dfnah, http://tinyurl.com/e4jfq

We've tested the system so far on 2 relatively inexperienced programmers
(as well as experienced, but the more interesting group is novices). The one
who hadn't done much programming at all (a little bit of VB, pre-university)
actually fared better IMO. This is probably because concurrency became
part of his standard toolbox of approaches.

I've placed the slides I've produced for Euro OSCON on Kamaelia here:
   * http://cerenity.org/KamaeliaEuroOSCON2005.pdf

The corrected URL for the whitepaper based on work now 6 months old (we've 
come quite a way since then!) is here:
   * http://www.bbc.co.uk/rd/pubs/whp/whp113.shtml

Consider a simple server for sending text (generated by a user typing into the 
server) to multiple clients connecting to a server. This is a naturally 
concurrent problem in various ways (user interaction, splitting, listening 
for connections, serving connections, etc). Why is that interesting to us? 
It's effectively a microcosm of how subtitling works. (I work at the BBC)

In Kamaelia this looks like this:

=== start ===
class ConsoleReader(threadedcomponent):
   def run(self):
      while 1:
         line = raw_input(">>> ")
         line = line + "\n"
         self.outqueues["outbox"].put(line)

Backplane("subtitles").activate()
pipeline(
    ConsoleReader(),
    publishTo("subtitles"),
).activate()
def subtitles_protocol():
    return subscribeTo("subtitles")

SimpleServer(subtitles_protocol, 5000).run()
=== end ===

The ConsoleReader is threaded to allow the use of the naive way of
reading from the input, whereas the server, backplane (a named splitter
component in practice), pipelines, publishing, subscribing, splitting,
etc are all single threaded co-operative concurrency.

A possible client for this text service might be:

pipeline(
    TCPClient("subtitles.rd.bbc.co.uk", 5000),
    Ticker(),
).run()

(Though that would be a bit bare, even if it does use pygame :)

The entire system is based around communicating generators, but we also
have threads for blocking operations. (Though the entire network subsystem
is non-blocking)

What I'd be interested in, is hearing how our system doesn't match with
the goals of the hypothetical concurrency system you'd like to see (if it
doesn't). The main reason I'm interested in hearing this, is because the
goals you listed are ones we want to achieve. If you don't think our system
matches it (we don't have process migration as yet, so that's one area)
I'd be interested in hearing what areas you think are deficient.

However, the way we're beginning to refer to the project is to refer to
just the component aspect rather than concurrency - for one simple
reason - we're getting to stage where we can ignore /most/ concurrency
issues(not all).

If you have any time for feedback, it'd be appreciated. If you don't I hope 
it's useful food for thought! 

Best Regards,


Michael
-- 
Michael Sparks, Senior R&D Engineer, Digital Media Group
Michael.Sparks at rd.bbc.co.uk, http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This e-mail may contain personal views which are not the views of the BBC.