python concurrency proposal

[Corey Coughlin]

OK, thanks for all this criticism, you've obviously taken some time 
here, guess I'll see if I can help clear some of this up....

Michael wrote:
<snip>
> 
> 
> On the surface of it, what you've described resembles Kamaelia[1] -
> specifically in the way used in the Axon Shell [2]. In other ways it
> differs wildy. I think it's interesting because this (or some of this)
> could be used as syntactic sugar for Kamaelia.
>     [1] http://kamaelia.sourceforge.net/Home
>     [2] http://kamaelia.sourceforge.net/AxonShell.html
> 
> The similarity is this: a pardef appears to be a function that can run,
> and be made to run in parallel with other functions. In Kamaelia we use
> generators to achieve precisely this.

Hey, you got it, great!

> However, given pythonic is a loaded term (beauty is in the eye of the
> beholder), I'd personally say that there's some elements of your syntax
> that suprise me, especially given the (IMO valid) approach of not being
> able to access inside the pardef.
> 
> Since there are some strong similarities between what you've written and
> Kamaelia it seems sensible to discuss them.
> 
> If you have your pardef:
> pardef <Name>(self, <par type>, arguments...):
>         self.send(<dest pardef>, <tag>, arguments)
>         self.receive(<tag>, arguments)
>         return arguments
>         yield arguments
> 
> This maps to this: (in your terms :) )
> 
> class <Name>(<par type>):
>     Inboxes = [ <tag>, <tag>, <tag> ] # Explicit is better than implicit
>     Outboxes = [ <tag>, <tag>, <tag> ] # There are defaults available...
>     def __init__(self, arguments ...):
>          // standard initialisation //
>          super(<Name>, self).__init__()
> 
>     def main(self):
>         // local
>         // Initialisation
>         while 1:
>             do stuff
>             yield //some value// (relinquish control to scheduler)
>         return
>         // We don't have the concept of a result, though this
>            would be useful, though we do have something
>            similar so this would be doable //
> 
> Inboxes and Outboxes are used as declarations to allow the baseclass
> (component) to initialise some datastructures for communications. These
> declarations are actually any iterable, and these days we tend to use
> dictionaries because it simplifies documentation.

Yes, it would map in a more detailed level to a class of some kind with 
the methods indicated.  Although, I'm imagining that for different <par 
type> base classes, the init and main loops would be implemented in 
different ways.  Different types of communication channels (shared 
memory copies, file based pickling, socket communication, and so on) 
would also have different implementations as well, beyond Inboxes and 
Outboxes.  This could make for some much more complicated base classes. 
   It also suggests some problems, notably that of coercing different 
messages from different types of base classes (i.e., suppose the 
standard library os and system classes get implemented as threads, but 
you still want to send and receive messages to and from them from 
processes and clustered processes).

> 
> An example here might control the behaviour of a sprite onscreen. So,
> suppose we have a sprite:
> 
<snip example>
> 
> Unlike your system, our components (your pardefs), don't know
> about each other and only talk to inboxes AND outboxes - these
> are connected at a higher level by something enclosing them.
> 
> Graphline(
>     Walker = SimpleSprite(walkerImage, 10,10),
>     WalkerLogic = WalkInASquare(10,10,400,400,50),
>     linkages = {
>        ("WalkerLogic", "position") : ("Walker", "position"),
>        ("WalkerLogic", "orientation") : ("Walker", "rotation"),
>     }
> )
> 
> This graphline can then be activated or run like any other component.

Yes, I've become a little worried about the casual way I'm pulling the 
pardefs together.  I did want to keep the definition fairly primitive, 
and then try to pull the jobs together using standard python objects 
like lists and dictionaries and such, but given that pympi and Kamaelia 
use more custom methods, it may be something to think about.  It might 
simplify the blocking/non-blocking behavior, which I'll get to soon...

> [[ For more complete examples, grab Kamaelia from the website,
>    download and run :) ]]
> 
> You'll note from the above that clearly there are aspects to it where
> your ideas could be used as syntactic sugar. Some of the benefits (like
> collapsing __init__ and main together) can be gained by using decorators
> in python 2.4.

I've actually been thinking that decorators may be the way to go to set 
the base class of the pardef, either that or have it be another class 
modifier.  Neither seems very ideal, but having it after self also seems 
kludgy.


> You'll also note there are some areas where your ideas increase parallel
> safety (such as when sending mutable data structures), at the expense on
> increasing the cost of communication.

Yes, the hope is that the ability to have different base classes that 
implement the same interface will allow people to experiment with 
different base implementations.  Sooner or later, the best solutions 
would migrate to the standard library, with luck.

> I'm not saying the above is the paragon of virtue (it isn't) - we're looking
> to see what's possible - but not rushing into creating syntax until we have
> an idea that the approaches work. That said, we are interested in ideas for
> syntaxes. At the moment I'd prefer these to be based on API enhancements
> rather than changes to python since I personally think that's an extreme
> approach until you know the change to the language will actually be
> beneficial.

Yeah, I guess I'm an extremist.  Ah well, I just like to think of what 
I'd ideally want, and see if I can get there.  Maybe I can't, or maybe 
nobody else shares my ideals, but I thought I'd toss it out and see what 
people think.

> ((In some respects we're held back from some improvements because we want
>   to stay supporting python on series 60 (which limits us to 2.2 late
>   features rather than things like decorators and PEP 342 enhancements).
>   We may drop that if necessary later or simply have a specialised version
>   there. This might explain the emphasis on ))
> 
> Regarding other points, of your suggestions though...
> 
> You allow the following to be equivalent initialisations:
> 
> h1 = vecadd(a[:500], b[:500], 'd')
> # vs
> h1 = vecadd()
> h1.veca = a[:500]
> h1.vecb = b[:500]
> h1.arrtype = 'd'
> 
> To me this looks dangerous. (as in likely to confuse and cause bugs)
> 
> What if vecadd is implemented as a thread, and can run concurrently with
> the main piece of code? (especially since you suggest reusing the same
> active object) This allows the possibility:
> 
> h1 = vecadd()
> h1.run() # does this background? Do we block? (appears to do the former)
> h1.veca = a[:500] # What happens, is h1 blocked until we get here?
> h1.vecb = b[:500]
> h1.arrtype = 'd'
> h1.veca = b[:500] # Uh oh, what's happening here?
> c = h1.result + h2.result # Um, what does this now mean ?
>
> were the values to veca queued? Were they overwritten? Would this be valid?
> To me it's completely non-obvious what this code means on reading it. It
> also *looks* like you're updating internal state of the object rather than
> an external interface. (It also has the implicit suggestion that you can
> read these values as well, which may or may not be valid)

Ugh.  Yeah, I go back and forth myself on the ability to change the 
arguments.  The only way it really works above is if the argument change 
  acts like a .send method, so it would probably have to act like the 
arguments are sitting in a queue until the next .run() is called. 
Probably the same for the .result, just take the next result from the 
last run call.  So the above sequence would hopefully just fail on the 
.run() where the arguments are undefined, but if they were defined, then 
the new arguments would go on a queue for the next run, I guess.  But 
yeah, it is probably the least 'pythonic' part of the proposal.  It just 
seems a little to valuable to give up.  Although I suppose it's possible 
to do, you could have to pass in all arguments through sends and 
receives.  Would that be less clunky or more clunky?  Hard to say.

> I think the core to of your idea is that your new suite really introduces
> a scheduler for generators and a means for communicating (you call them
> tags). What you've done here is also make the arguments for creation mutable
> which becomes confusing.

Actually, I don't think I introduced a scheduler, and I sure don't want 
to limit things to generators.  I kind of figured that a scheduler would 
be something a little more advanced that might be added in a library at 
some point.  I just wanted to see if people liked the primitive first off.

> I'm not trying to discourage you, I like the ideas, and would like to see
> you expand them more since they interest me, but you have some other
> inconsistencies. For example, you say here:
> 
> Remember your statement that pardefs don't inherit any data from the global
> scope. Data is only passed in through receive statements and the arguments.
> 
> In practice however, you also have in your examples this:
>         updef = not (isinstance(up, int) or isintance(up, float))
>         downdef = not (isinstance(down, int) or isintance(down, float))
>         rightdef = not (isinstance(right, int) or isintance(right, float))
>         leftdef = not (isinstance(left, int) or isintance(left, float))
> 
> In these lines you have passed over global scope data - specifically
> int and float.

Well, the old preferred method was to import types and compare to those, 
but I thought that comparing to the builtins was the shiny new way to do 
things.  I don't do this stuff very often, so I may have gotten this 
wrong, though.  But I do assume that a pardef does get a clean set of 
built in functions, as you would get in any module without imports.

> Unlike your approach (syntax first), we've created a large (growing) number
> of components (like your pardefs) which have been put together for various
> systems which have varying levels of concurrency which are probably a useful
> testbed for testing your ideas:
> 
>     * http://kamaelia.sourceforge.net/Components.html
> 
> Example systems created using this vary from a "everything broadcast" PVR
> for radio [*], a networked audio mixer matrix, simple games, a presentation
> tool (probably including soon a live video mixer to go with the video
> playback), topology viewing tools, a simple "paint" program, etc)
> 
>    * Actually a box for creating a record of transmission, which
>      is slightly different, but the former description is more
>      accessible if slightly inaccurate.)
> 
> If you're interested in prototyping your ideas in python, you can simulate
> some of your ideas using decorators. Something that might help you with
> prototyping your ideas is our tutorial for Kamaelia, which is a "build your
> own core" system type tutorial. It might also help show that your pardefs
> are very similar to python's generators. It can be found here:
>     * http://kamaelia.sourceforge.net/MiniAxon/

Yes, you have been busy.  It is a very interesting system.

> In many respects, I feel that the API we have still isn't "100% pythonic" as
> many would call it, but it does try to be unsurprising and consistent. You
> could say we're aiming for pythonic - though personally I favour easy and
> unsurprising as more concrete, less abstract goals - even if it's not there
> yet. (portability of ideas to other languages is important to me, which
> again is another reason for an API based view rather than syntax).
> 
> If you're willing to take a look at it, and make suggestions as to how your
> ideas might fit, (or what you think is dumb :-) I'd welcome it.

I'm not sure, really, I'm still kind of confused as to the depth of your 
system.  It does look strongly generator based, so it should be possible 
  to port most of the algorithms you're using to this proposed system 
(if I ever implement it) (and I really should mention that a .restart 
method would really be useful for yield-ing pardefs, I keep forgetting 
to mention that) but I'm not sure how much further Kamaelia goes.  It 
sounds like you have networking 'pipes' going that allow you distribute 
jobs across a network, but it's not completely clear to me.  If that's 
the case, then it should be possible to encapsulate that into a Kamaelia 
base class for this proposal so you could keep that functionality as 
well.  That might require some extra base-class specific methods, but I 
have no problem with that.  Like I said, I imagine this as a fairly 
minimal interface, no problem supplementing it if you like.
	But anyway, I'm beginning to ramble, but thanks for all the great 
feedback, and a happy new year to you as well!  :)


        ----- Corey

> *Especially* if it simplifies the system (*or* the way it's used).
> 
> Finally though, as I say above,I'm not trying to discourage you, I like
> the ideas, and would like to see you expand them more since they interest
> me, and like you I think this is an area that needs work. I would suggest
> playing with the ideas though and testing them against systems before
> writing a PEP though. (real/useful systems rather than contrived ones! -)
> 
> Best Regards & Happy New Year,
> 
> 
> Michael.
> --
> Michael.Sparks at rd.bbc.co.uk, http://kamaelia.sourceforge.net/ 
> British Broadcasting Corporation, Research and Development 
> Kingswood Warren, Surrey KT20 6NP 
>  
> This message (and any attachments) may contain personal views
> which are not the views of the BBC unless specifically stated.
>