[IPython-dev] Musings: syntax for high-level expression of parallel (and other) execution control

[Darren Dale]

On Sun, Sep 6, 2009 at 9:06 AM, Darren Dale<dsdale24 at gmail.com> wrote:
> Hi Prabhu,
>
> On Sun, Sep 6, 2009 at 3:14 AM, Prabhu
> Ramachandran<prabhu at aero.iitb.ac.in> wrote:
>> On 09/06/09 11:52, Fernando Perez wrote:
>>>
>>> But don't beat yourself up: while there's nothing complex about what
>>> we're talking about, it is a slightly unusual usage of the language,
>>> so it's natural to do a double-take with it.  I have the benefit of
>>> having worried about this problem for a long time, but it took me
>>> *many* tries to understand how to fit the pieces together.  And I had
>>> the advantage of lots of help along the way:
>>
>> [...]
>>>
>>> - and finally, John Siracusa's review at Ars Technica about Apple's
>>> work with anonymous blocks and Grand Central Dispatch make the whole
>>> thing click.
>>>
>>> As you can see, if you're slow for taking a day to put it together,
>>> there's simply no hope for me: it took me almost 2 years, and I needed
>>> the help of some of the very brightest people in the python world to
>>> push me along.  They are the ones who did all the thinking and deserve
>>> the credit, I was just thick enough never to understand the ideas
>>> until now!
>>
>> Thanks for the interesting links and thread.  Just FYI, last year I had
>> occasion to solve, relatively elegantly, a set of pretty sticky problems for
>> mayavi2 using decorators and generators.  See for example:
>>
>> https://svn.enthought.com/enthought/browser/Mayavi/trunk/enthought/mayavi/tools/show.py
>> https://svn.enthought.com/enthought/browser/Mayavi/trunk/enthought/mayavi/tools/animator.py
>>
>> They allow us to do relatively simple but neat things very elegantly. Until
>> you use decorators you often don't realize how convenient they can be.  In
>> particular the pattern used in animator.py shows how convenient the
>> combination of a UI dispatch mechanism plus a generator is.  All a
>> programmer needs is to inject a yield suitably and the rest is automatic.
>>
>> The Kamaelia project (http://www.kamaelia.org) is also very interesting for
>> its use of generators, microprocesses, components and very specifically
>> concurrency.  They have a very neat model for exactly this and a nice though
>> slightly elaborate tutorial showing how you can build their core library
>> from these ideas.  If you have the time it is very interesting.
>>
>> My humble contribution to spending two more hours of your time. ;-)
>
> Thanks for the pointer. The Pipeline and Graphline reminds me of
> LabView, or rather how I would prefer to develop labview-like
> applications in python.
>
> I had to look in the older document structure to find a discussion on
> concurrency, here it is for anyone who is interested:
> http://www.kamaelia.org/Docs/Axon/Axon.Microprocess.html

They talk a lot about microprocesses, and in this pdf (1) they mention
multi-core computers, but I think their terminology unfortunately
confuses processes with threads. From the looks of their trunk, they
are using the threading package for "concurrency". They are aware of
the GIL, however (2).

(1) http://www.google.com/url?sa=t&source=web&ct=res&cd=2&url=http%3A%2F%2Fwww.kamaelia.org%2Ft%2FTN-LightTechnicalIntroToKamaelia.pdf&ei=x7SjStviLomJ8QbgybXSDw&usg=AFQjCNH8l1dPhesP9svjU_eBvAmzTgYjkQ&sig2=43cMT10EIYC715NMy4-E1Q

(2) http://groups.google.com/group/kamaelia/browse_thread/thread/23b203b183a0c3e4