Adding a Par construct to Python?

[Rhodri James]

On Wed, 20 May 2009 09:19:50 +0100, <jeremy at martinfamily.freeserve.co.uk>  
wrote:

> On 20 May, 03:43, Steven D'Aprano
> <ste... at REMOVE.THIS.cybersource.com.au> wrote:
>> On Tue, 19 May 2009 03:57:43 -0700, jeremy wrote:
>> > As I wrote before, concurrency is one of the hardest things for
>> > professional programmers to grasp. For 'amateur' programmers we need  
>> to
>> > make it as simple as possible,

Here, I think, is the fatal flaw in your plan.  As Steven pointed out,
concurrency isn't simple.  All you are actually doing is making it
easier for 'amateur' programmers to write hard-to-debug buggy code,
since you seem to be trying to avoid making them think about how to
write parallelisable code at all.

> I *do* actually know a bit about concurrency and would never imply
> that *any* for loop could be converted to a parallel one. The
> intention of my remark "with my suggestion they could potentially get
> a massive speed up just by changing 'for' to 'par' or 'map' to
> 'pmap'." is that it could be applied in the particular circumstances
> where there are no dependencies between different iterations of the
> loop.

If you can read this newsgroup for a week and still put your hand on
your heart and say that programmers will check that there are no
dependencies before swapping 'par' for 'for', I want to borrow your
rose-coloured glasses.  That's not to say this isn't the right solution,
but you must be aware that people will screw this up very, very
regularly, and making the syntax easy will only up the frequency of
screw-ups.

> This shows why the sync event is needed - to avoid  race conditions on
> shared variables. It is borrowed from the BSP paradigm - although that
> is a distibuted memory approach. Without the sync clause, rule 5 would
> just be the standard way of defining a parallelisable loop.

Pardon my cynicism but sync would appear to have all the disadvantages
of message passing (in terms of deadlock opportunities) with none of
advantages (like, say, actual messages).  The basic single sync you put
forward may be coarse-grained enough to be deadlock-proof, but I would
need to be more convinced of that than I am at the moment before I was
happy.

> P.S. I have a couple of additional embellishments to share at this
> stage:
[snip]
> 2. Scope of the 'sync' command. It was pointed out to me by a
> colleague that I need to define what happens with sync when there are
> nested par loops. I think that it should be defined to apply to the
> innermost par construct which encloses the statement.

What I said before about deadlock-proofing?  Forget it.  There's hours
of fun to be had once you introduce scoping, not to mention the fact
that your inner loops can't now be protected against common code in the
outer loop accessing the shared variables.

-- 
Rhodri James *-* Wildebeeste Herder to the Masses