[pypy-dev] STM/AME for CPython?

[Bengt Richter]

On 04/13/2012 06:11 AM Armin Rigo wrote:
> Hi Bengt,
>
> Actually I think we need foremost a new name.  This is not about doing
> great things in any of the domains you list.  It has no obvious
> connexion to any separation of metadata, security, capability,
> distributed update, mobile, or any other specific environment. This is
> merely about adding a new one-liner API usable by programs in order to
> behave as if they were run serially, but actually run on multiple
> cores.  So even using the name "STM" is rather wrong and confusing:
> STM is only one possible implementation.
>
> This is described explicitly by the OCM project:
> http://ocm.dreamhosters.com/ .  The goal I am pursuing is slightly
> different than OCM so using that name is not right either, but at
> least it is at a similar level.
>
> On a different note, I found an implemention for C/C++ programs using
> all the tricks of paging under Linux:
> http://plasma.cs.umass.edu/emery/grace .  Does not work at all on
> CPython, where reference counting is enough to kill it: almost no page
> will be read-only.
A thought experiment: what if you squandered pages by allocating a buddy-page
for each page with refcount variables, and did your reference counting
at the same offset in the buddy page, leaving the original untouched
as far as the ref counting goes? And then don't write protect the
buddy/refcount page, but do for the original, so you can use
the page fault mechanism to detect data writes other than reference count updates.
... (sort of what I was getting at referring to issues of separation of data and metadata).
Of course you could do better space-wise by putting a level of indirection in the ref counting,
and point to refcount values allocated packed in the buddy/refcount pages (which wouldn't be
"buddy" 1:1 any more). Similar considerations might apply to other metadata.
Haven't thought about how garbage collection would be affected, but I gotta stop now ;-)

>
> I'm writing down these papers in a file in the duhton repository.  It
> would be cool if someone reviews them and comes up with a great
> general name :-)
>
Much of the discussion reminds me of CPU pipelines with out-of-order
and speculative execution and discarding of results from paths not taken
when calculated conditions become available. So maybe a name like
     Speculative Parallel Software Execution Merging (SPSXM?) (please improve ;-)

But if I understand you right, that's an implementation detail underneath
your "one-liner API" anyway, and not your primary concern?
(until you get to optimizing, IWG ;-)

But do you really need a new one-liner for python? I.e, couldn't you use
a decorator to transmogrify a function into just about anything, like to create atomic
functions, optionally also asynchronous? (wouldn't surprise me if this exists already).

But given duhton[1], it is maybe not python per se but rpython
that you want to make a one-liner for? I obviously haven't read enough, sorry ;-/
(OTOH I may represent a few other in that).

BTW, will your solution enable us to create atomic properties using descriptors and all that?

Regards,
Bengt Richter
PS. Nit: I wasn't familiar with the acronym AME (Automatic Mutual Exclusion). Might be good to
spell it out on first use in your docs, as it didn't come up in wikipedia ;-)
[1] BTW, why not -'thon'? ;-)