[Python-ideas] Python and Concurrency

[Ron Adam]

Jim Jewett wrote:
> On 3/22/07, Ron Adam <rrr at ronadam.com> wrote:
>> Josiah Carlson wrote:
>> > Ron Adam <rrr at ronadam.com> wrote:
>> >> Josiah Carlson wrote:
> 
>> >>> On the upside, this is possible (change the PyObject_HEAD macro,
>> >>> PyINCREF, PyDECREF, remove the GIL), but the amount of work
> 
> The zillions of details are the same details you have to consider when
> writing a concurrent database.  Having python store its objects
> externally and retrieve them when needed (vs allocating memory and
> pointers) would be a huge one-time loss, but it might eventually be
> worthwhile.
> 
> PyPy calls that "external database" an "object_space", so the language
> support is already there, and still will be when the hardware makes it
> worthwhile.
> 
> 
>> (1)
>>     xyz = vector(r)
>>     forall coords as c:     # parallel modify coords 'inplace' with body
>>         c = rotate(c, xyz)
>>
>>     * 'with' syntax form because 'c' does not outlive the body.
> 
> 
> Why not just keep using
> 
>    xyz = vector(r)
>    rotate(xyz)
> 
> and let the compiler take care of it?

Just how much should the 'python' compiler do?

And what about dynamic modeling where things change according to input?

(Not just thinking of graphics.)


> At most, we would want a way of marking objects as "read-only" or
> callables as "instant", so that the compiler would know it doesn't
> have to worry about the definition of "len" changing mid-stream.  (Ron
> suggests something similar at the end of the message, and Talin's
> Transactional metaclass is related.)

Would it be possible to have a container for which it's contents are read 
only?  (as a side effect of being in the container)  Then individual items 
would not need their own read only attributes.


A few other thoughts...

Possibly ... when a module is first imported or ran nothing needs to be 
marked read only.  Then when execution falls off the end, *everything* 
existing at that point is marked read only. If the module was run from as a 
script, then a main function is executed.  These could be optional settings 
that could be turned on...

     from __far_future__ import multi-processing
     __multi-processing__ = True
     __main__ = "my_main"

So then there would be an initiation first pass followed by an execution 
phase. In the initiation phase it's pretty much just the way things are now 
except you can't use threads.  In the execution phase you can use threads, 
but you can't change anything created in the initiation phase.

(Other controls would still be needed of course.)


>> >> * Generators with the semantics of calculating first and waiting on 
>> 'yield'
>> >> for 'next', so the value is immediately returned. (depends on CPU 
>> load)
> 
> On its own, this just makes response time worse.

Yes, you really wouldn't use these for very simple counters.  For more 
complex things, the calling overhead becomes a much smaller percentage of 
the total, and it would have a bigger effect on smoothing out response time 
rather than hurting it.


> That said,
> generators are also callables, and the compiler might do a better job
> if it knew that it wouldn't have to worry about external
> redefinitions.
> 
> There may also be some value in some sort of "idletasks" abstraction
> that says "hey, go ahead and precompute this, but only if you've got
> nothing better to do."

Having a way to set process priority may be good.  (or bad)


>Garbage collection could certainly benefit
> from this, and translating (or duplicating) string representations
> into multiple encodings.
> 
>> In the video he also talked about avoiding locks.  I was thinking a more
>> limited function object (for concurrent uses only) might be used that 
>> has:
> 
>>     * no global keyword
>>     * only access to external immutable objects
> 
> Or at least, it doesn't mutate them itself, and it doesn't promise to
> use newer versions that get created after the call begins.

The difficulty is that even a method call on a global (or parent scope) 
object can result in it being mutated.  So you are either back to using 
locks and/or transactions on everything.


>>     * no closures.
> 
> This level of information should be useful.
> 
> But in practice, most of my functions already meet this definition.

Mine too.

Ron



> (In theory, they access builtins that could be replaced, etc.)  I
> wouldn't want to mark them all by hand.  I'm still inclined to trust
> the compiler, and just accept that it may eventually be the PyPy
> translator rather than the CPython interpreter.
> 
> -jJ
> 
>